another ordering question

[calischs]

Hi there, First of all, great job on floweaver! I hadn't seen it until yesterday, and it was reasonably easy to get up and running with it. The results have been encouraging so far!

I'm trying to use floweaver to make a similar sankey to the fantastic one in "The efficient use of energy: Tracing the global flow of energy from fuel to service", Jonathan M. Cullen, Julian M. Allwood. Below is the original:

I thought I'd start by recreating the original. Here is my first attempt:

I'm learning how to use floweaver, so pardon the messiness. (happy to post the csv if it is useful)

dataset = Dataset.from_csv('worldwide-energy-use-data.csv')
size = dict(width=3000, height=1200)
primary_sources = ['Renewable','Nuclear', 'Coal','Oil', 'Gas', 'Biomass', ]
conversion_devices = ['1Motion','2Other','3Heat'] #keys have numbers to enforce ordering: https://github.com/ricklupton/floweaver/issues/65
passive_systems = ['Vehicle','Factory','Building']
materials = ['Chemical','Steel','Mineral','Paper','Food','Aluminum','Other']
final_services = ['Passenger Transport','Freight Transport','Structure', 'Sustenance','Hygiene','Thermal Comfort','Communication','Illumination']
#extra labels for final services
final_services_outputs = {
    'Passenger Transport':'23e12 passenger-km',
    'Freight Transport':'46e12 tonne-km',
    'Structure':'15e9 MPa^2/3m^3',
    'Sustenance':'28e18 J (food)',
    'Hygiene':'1.5e12 m^3K (water)\n 2.8e18 Nm (work)',
    'Thermal Comfort':'30e15 m^3K (air)',
    'Communication':'280e18 (bytes)',
    'Illumination':'480e18 lm s'
}
nodes = {
    'Primary Energy': ProcessGroup(primary_sources),
    'Electricity':ProcessGroup(['Electricity']),
    'Conversion': ProcessGroup(conversion_devices),
    'Passive Systems':ProcessGroup(passive_systems),
    'Material':ProcessGroup(['Material']),
    'Final Services':ProcessGroup(final_services),
    'Nonmaterial1':Waypoint(title=''),
    'Nonmaterial2':Waypoint(title=''),
    'Direct Use':Waypoint(),
    'Electrical conversion devices':Waypoint(),
    'Direct Use conversion devices':Waypoint(),
    'Factory materials':Waypoint(),
    'System type':Waypoint(),

}
ordering = [
    ['Primary Energy'],
    ['Electricity','Direct Use'],
    ['Electrical conversion devices','Direct Use conversion devices'],
    ['Conversion'],
    ['System type'],
    ['Passive Systems'],  
    ['Factory materials','Nonmaterial1'],
    ['Material','Nonmaterial2'],  
    ['Final Services'],      
]
bundles = [
    Bundle('Primary Energy', 'Electricity'),
    Bundle('Primary Energy', 'Conversion', waypoints=['Direct Use','Direct Use conversion devices']),
    Bundle('Electricity', 'Conversion',waypoints=['Electrical conversion devices']),
    Bundle('Conversion', 'Passive Systems',waypoints=['System type']),
    Bundle('Passive Systems','Material',waypoints=['Factory materials']),
    Bundle('Material','Final Services'),
    Bundle('Passive Systems','Final Services',waypoints=['Nonmaterial1','Nonmaterial2']),
]

palette = {
    #source partition palette entries
    'Oil': 'brown',
    'Biomass': 'lightgreen',
    'Coal': 'black',
    'Gas': 'orange',
    'Nuclear': 'purple',
    'Renewable': 'green',
    'Electricity': 'gold',
    '3Heat': 'crimson',
    '1Motion': 'deepskyblue',
    '2Other': 'darkgrey',
    'Building':'forestgreen',
    'Factory':'orchid',
    'Vehicle':'deepskyblue',
    'Material':'orchid',
}

nodes['Primary Energy'].partition = Partition.Simple('process', primary_sources)
nodes['Conversion'].partition = Partition.Simple('process', [('Motion',['1Motion']),('Other',['2Other']),('Heat',['3Heat'])])
nodes['Passive Systems'].partition = Partition.Simple('process', passive_systems)
nodes['Final Services'].partition = Partition.Simple('process', [(k+'\n'+v,[k]) for k,v in final_services_outputs.items()])
#nodes['Nonmaterial1'].partition = dataset.partition('source') 
nodes['Nonmaterial1'].partition = Partition.Simple('source', [(' ',['Vehicle']),('  ',['Building'])]) #stupid, to eliminate labels
#nodes['Nonmaterial2'].partition = dataset.partition('source') 
nodes['Nonmaterial2'].partition = Partition.Simple('source', [(' ',['Vehicle']),('  ',['Building'])]) #stupid, to eliminate labels

#nodes['Direct Use'].partition = dataset.partition('source') #break direct use of source fuels by source
nodes['Direct Use'].partition = Partition.Simple('source', [(' ',['Oil']),('  ',['Biomass']),('   ',['Gas']),('    ',['Coal'])]) #stupid, to eliminate labels

nodes['Electrical conversion devices'].partition = dataset.partition('type')
nodes['Direct Use conversion devices'].partition = dataset.partition('type') 
nodes['Factory materials'].partition = dataset.partition('type')
nodes['System type'].partition = dataset.partition('type')

sdd = SankeyDefinition(nodes, bundles, ordering, flow_partition=dataset.partition('source'))
weave(sdd, dataset, palette=palette).to_widget(**size,margins=dict(left=200, right=400),align_link_types=False).auto_save_png('worldwide-energy-sankey-v1.png')

My main question has to do with ordering. Obviously, the blue 'Vehicle' flow at the right that dips down and comes back up is suboptimal. We would like to have it above the 'material' flow. The 'material' and 'building' flows are constructed using Waypoints in this line:

Bundle('Passive Systems','Final Services',waypoints=['Nonmaterial1','Nonmaterial2']),

Because of this, I can only figure out how to put both above, or both below, as pictured. Any ideas? Is there a better way to do this kind of passthrough?

I also have a couple small questions about formatting labels (sorry if they should be separate issues each):

1. I'd like to shift the text to be vertically and horizontally centered on the nodes (e.g. by using text-align and vertical-align). I've tried this using the snippet below, but it doesn't work:

   %%html
   <style>
   .sankey .node {
   font-size: 36px;
   text-align: center;
   }
   </style>

   Any ideas to shift these labels?

2. I'd also love to use MathJax in the text labels as I do with matplotlib (e.g. for the units at the far right). Any chance this works? I tried using $ signs, without any luck.

3. Is there a way to turn off the vertical lines at each node?

Thanks, sam

[calischs]

I ended up finding a workaround for the main question in this issue by introducing a dummy node into the dataset, called "Vehicle2", and assigning all the flow from "Vehicle" to the final services through this node. Then, I could apply whatever order I wanted.

dataset = Dataset.from_csv('worldwide-energy-use-data.csv')
size = dict(width=3000, height=1200)
primary_sources = ['Oil','Biomass', 'Gas','Coal', 'Nuclear','Renewable', ]
conversion_devices = ['1Motion','2Heat','3Other',] #keys have numbers to enforce ordering: https://github.com/ricklupton/floweaver/issues/65
passive_systems = ['Vehicle','Factory','Building']
materials = ['Chemical','Steel','Mineral','Paper','Food','Aluminum','Other']
final_services = ['Passenger Transport','Freight Transport','Structure', 'Sustenance','Hygiene','Thermal Comfort','Communication','Illumination']
#extra labels for final services
final_services_outputs = {
    'Passenger Transport':'23e12 passenger km',
    'Freight Transport':'46e12 tonne km',
    'Structure':'15e9 MPa^2/3 m^3',
    'Sustenance':'28e18 J (food)',
    'Hygiene':'1.5e12 m^3K (water)\n 2.8e18 N m (work)',
    'Thermal Comfort':'30e15 m^3 K (air)',
    'Communication':'280e18 bytes',
    'Illumination':'480e18 lm s'
}
nodes = {
    'Primary Energy': ProcessGroup(primary_sources),
    'Electricity':ProcessGroup(['Electricity']),
    'Conversion': ProcessGroup(conversion_devices),
    'Passive Systems':ProcessGroup(passive_systems),
    'Material':ProcessGroup(['Material']),
    'Final Services':ProcessGroup(final_services),
    'Vehicle2':ProcessGroup(['Vehicle2']),
    'Nonmaterial1':Waypoint(title=''),
    'Nonmaterial2':Waypoint(title=''),
    'Direct Use':Waypoint(),
    'Electrical conversion devices':Waypoint(),
    'Direct Use conversion devices':Waypoint(),
    'Factory materials':Waypoint(),
    'System type':Waypoint(),

}
ordering = [
    ['Primary Energy'],
    ['Direct Use','Electricity',],
    ['Direct Use conversion devices','Electrical conversion devices',],
    ['Conversion'],
    ['System type'],
    ['Passive Systems'],  
    ['Vehicle2','Factory materials','Nonmaterial1'],
    ['Material','Nonmaterial2'],  
    ['Final Services'],      
]
bundles = [
    Bundle('Primary Energy', 'Electricity'),
    Bundle('Primary Energy', 'Conversion', waypoints=['Direct Use','Direct Use conversion devices']),
    Bundle('Electricity', 'Conversion',waypoints=['Electrical conversion devices']),
    Bundle('Conversion', 'Passive Systems',waypoints=['System type']),
    Bundle('Passive Systems','Material',waypoints=['Factory materials']),
    Bundle('Material','Final Services'),
    Bundle('Passive Systems','Vehicle2'),
    Bundle('Vehicle2','Final Services'),    
    Bundle('Passive Systems','Final Services',waypoints=['Nonmaterial1','Nonmaterial2']),
]

palette = {
    #source partition palette entries
    'Oil': 'brown',
    'Biomass': 'lightgreen',
    'Coal': 'DimGray',
    'Gas': 'orange',
    'Nuclear': 'purple',
    'Renewable': 'green',
    'Electricity': 'gold',
    '1Motion': 'deepskyblue',
    '2Heat': 'crimson',
    '3Other': 'darkgrey',
    'Building':'forestgreen',
    'Factory':'orchid',
    'Vehicle':'deepskyblue',
    'Vehicle2':'deepskyblue',
    'Material':'orchid',
}

nodes['Primary Energy'].partition = Partition.Simple('process', primary_sources)
nodes['Conversion'].partition = Partition.Simple('process', [('Motion',['1Motion']),('Heat',['2Heat']),('Other',['3Other']),])
nodes['Passive Systems'].partition = Partition.Simple('process', passive_systems)
nodes['Final Services'].partition = Partition.Simple('process', [(k+'\n'+v,[k]) for k,v in final_services_outputs.items()])
#nodes['Nonmaterial1'].partition = dataset.partition('source') 
nodes['Nonmaterial1'].partition = Partition.Simple('source', [(' ',['Vehicle']),('  ',['Building'])]) #hack to eliminate labels
#nodes['Nonmaterial2'].partition = dataset.partition('source') 
nodes['Nonmaterial2'].partition = Partition.Simple('source', [(' ',['Vehicle']),('  ',['Building'])]) #hack to eliminate labels

#nodes['Direct Use'].partition = dataset.partition('source') #break direct use of source fuels by source
nodes['Direct Use'].partition = Partition.Simple('source', [(' ',['Oil']),('  ',['Biomass']),('   ',['Gas']),('    ',['Coal'])]) #hack to eliminate labels

nodes['Electrical conversion devices'].partition = dataset.partition('type')
nodes['Direct Use conversion devices'].partition = dataset.partition('type') 
nodes['Factory materials'].partition = dataset.partition('type')
nodes['System type'].partition = dataset.partition('type')
nodes['Vehicle2'].partition = Partition.Simple('type',[(' ',['Vehicle2'])]) #hack to remove "Vehicle2" label, though a line still shows up for some reason...

sdd = SankeyDefinition(nodes, bundles, ordering, flow_partition=dataset.partition('source'))
weave(sdd, dataset, palette=palette).to_widget(**size,margins=dict(left=200, right=400),align_link_types=False).auto_save_png('worldwide-energy-sankey-v1.png')

[ricklupton]

Hi, I think you can do this without adding any fake nodes into the actual dataset.

The problem is that you want the flows coming from Factory to go through the Factory materials waypoint on their way from Passive Systems to Final Services, but you want the flows from Vehicle and Building to go directly.

The simplest solution is probably to split the Passive Systems subgroup into 3. Something like:

nodes = {
    # instead of 'Passive Systems':ProcessGroup(passive_systems),
    'Vehicles': ProcessGroup(['Vehicles']),
    'Factory': ProcessGroup(['Factory']),
    'Building': ProcessGroup(['Building']),
}

ordering = [
    #...
    ['Vehicles', 'Factory', 'Building'],
    ['VehiclesWaypoint1', 'Factory materials','BuildingsWaypoint1'],
    ['VehiclesWaypoint2', 'Material', 'BuildingsWaypoint2'],  
    ['Final Services'],      
]
bundles = [
    # instead of Bundle('Passive Systems','Material',waypoints=['Factory materials']),
    Bundle('Vehicles', 'Passive Systems', waypoints=['VehiclesWaypoint1', 'VehiclesWaypoint2']),
    Bundle('Factory', 'Material', waypoints=['Factory materials']),
    Bundle('Material','Final Services'),
    Bundle('Buildings', 'Passive Systems', waypoints=['VehiclesWaypoint1', 'VehiclesWaypoint2']),
]

Hope that makes sense.

[ricklupton]

Thanks for the nice example -- would you like to add it to the cookbook when you get it working? https://github.com/ricklupton/floweaver/tree/master/docs/cookbook

For your other questions, shifting the text with CSS is probably difficult because the position is calculated in pixels relative to the size of the node. You could try playing with transform/translate properties, I'm not sure if that would work.

MathJax -- I don't think this is possible at the moment. I created an issue to keep track of this: https://github.com/ricklupton/ipysankeywidget/issues/43

Hiding lines: Yes you should be able to do this with CSS. Try using the DOM inspector to see what the structure is -- something like

.sankey .node line { display: none; }

[ricklupton]

One more thing -- you commented in your code "keys have numbers to enforce ordering: https://github.com/ricklupton/floweaver/issues/65" but that applies only to the flow partition (set with SankeyDefinition(... flow_partition=X). The partitions for ProcessGroups end up in the order specified.

Now that I say this, I'm not actually sure the problem in #65 happens for any good reason -- the sorting of links sharing a source or target could probably preserve the order given in the partition. Need to check in the code if there's a reason why this wouldn't work.

[calischs]

Sorry for the delayed reply, and thanks for all the info.

Your strategy for the strand ordering makes sense -- I'll work on updating the example to use it. Happy to add it to the cookbook when it's finished. :) (though some flows are estimated from the original figure, so it should probably have a disclaimer)