Disable sub-sectors in Atlas+Refinery

[wouterterlouw]

In https://github.com/quintel/refinery/issues/52 the following was proposed to disable the Metal sub-sector.

I think a good compromise would be a hybrid of the two suggestions; keep the metal sub-sector nodes and edges, but set them all to zero. We could do this within Atlas+Refinery when you set - has_metal = false.

@antw Can you implement this?

[wouterterlouw]

@ChaelKruip Can you have a look on this issue and let us know if you approve with this approach?

[ChaelKruip]

@ChaelKruip Can you have a look on this issue and let us know if you approve with this approach?

:+1:

[wouterterlouw]

@antw What is the status of this issue?

[antw]

I have pushed Atlas and ETSource branches so that you can try out this feature for yourself; you need to use both together:

• quintel/atlas@disable-subsectors
• quintel/etsource@metal-subsector

The ETSource branch moves the "metal" nodes into a new subdirectory: nodes/industry/metal so that we can differentiate them from other industry nodes. The Atlas branch introduces the behaviour that has_metal = false should disable nodes in the "industry.metal" sub-sector; we can make this more general – supporting other sub-sectors – if you like the implementation.

[wouterterlouw]

@antw Looks nice! :+1:

During testing I run into the following:

• The metal industry has also some converters in the energy part of the graph:
  • energy_cokesoven_consumption_coal_gas.converter.ad
  • energy_cokesoven_transformation_coal.converter.ad
  • energy_steel_blastfurnace_bat_transformation_cokes.converter.ad
  • energy_steel_blastfurnace_current_transformation_cokes.converter.ad
  • energy_steel_hisarna_transformation_coal.converter.ad
  • energy_distribution_coal_gas.ad
  • energy_distribution_cokes.converter.ad
  • energy_import_coal_gas.converter.ad
  • energy_export_coal_gas.converter.ad
• Running ETengine with the two branches (and new csv files generated using the industry analysis without importing the metal industry data, imported from the quintel/etdataset@metal-subsector branch gives the following error:

Refinery::IncalculableGraphError at /data/latest/area
Insufficient data to calculate 5 items: 

  * EdgeDemand for #<Refinery::Edge :industry_other_metals_burner_network_gas -:useable_heat-➤ :industry_other_metals_process_heat_useable_heat (type=flexible)>
  * NodeDemand for #<Refinery::Node key=:industry_other_metals_burner_network_gas>
  * EdgeDemand for #<Refinery::Edge :industry_other_metals_process_heat_useable_heat -:not_defined-➤ :industry_other_metals_production (type=)>
  * NodeDemand for #<Refinery::Node key=:industry_other_metals_process_heat_useable_heat>
  * NodeDemand for #<Refinery::Node key=:industry_other_metals_production>

[antw]

The metal industry has also some converters in the energy part of the graph

In which case I think we'll need something more flexible than placing the node files into subdirectories; perhaps we could create a new node group – "metal" – and use the has_metal = false setting to disable any node belonging to that group...

Running ETengine with the two branches [..] using the industry analysis without importing the metal industry data [..] gives the following error:

That suggests some nodes aren't being disabled, which is odd. I'll look into it now.

[antw]

I have pushed updated ETSource and Atlas branches. In ETSource I moved the metal nodes back into the main "industry" directory, and added them to a new "metal" group (and also the energy nodes listed above). Atlas has been updated so that has_metal = false disables nodes in that group.

As for the IncalculableGraphError, I think you forgot to set has_metal = false in the dataset files; the error is there when the value is true, and disappears when set to false.

[wouterterlouw]

@antw It looks like the ETsource branch is not updated yet...

[antw]

@wouterterlouw My bad; pushed.

[wouterterlouw]

@antw Thanks!

As for the IncalculableGraphError, I think you forgot to set has_metal = false in the dataset files; the error is there when the value is true, and disappears when set to false.

I think it has to do with the input efficiencies. The industry_other_metals_burner_network_gas does have three (?) different carriers as input. If I set all the shares in the metal industry to zero, the input efficiencies cannot be defined. You can have a look using the quintel/etsource@metal-subsector-updated-dataset branch.

[ChaelKruip]

Related: https://github.com/quintel/etsource/issues/326

[antw]

You can have a look using the quintel/etsource@metal-subsector-updated-dataset branch.

@wouterterlouw I can't seem to reproduce the incalculable graph when using this branch with the "disable-subsectors" Atlas branch. I can if I set has_metal to true, or if I run the rake debug task using the Atlas master branch.

Since your earlier post says that you tried this in ETEngine, that might explain the issue since ETE is linked to the Atlas master branch. If you want to try using a local copy of the "disable-subsectors" branch, you need to change the Gemfile in the ETEngine directory;

Remove this:

gem 'atlas', ref: '4885239', git: 'git@github.com:quintel/atlas.git'

Replace it with this:

gem 'atlas', path: 'path/to/your/atlas/clone'

Then run bundle.

[wouterterlouw]

@antw :+1: With the replacement in the Gemfile ETengine works fine!

[wouterterlouw]

@antw Can we merge this implementation or do you want to make it more general before?

@ChaelKruip What do you think?

[antw]

I would like to make it more general – so that it is possible to disable any node group without it having to be hard-coded in Atlas – but that will take a while longer, and will involve some tweaks to ETEngine also. I don't think it's a good idea doing that so close to EC2013, so how about we merge this as-is, and I'll improve it next week once the convention is over?