More useable CSVwriter for Merit wanted

[ChaelKruip]

From #103:

Perhaps it would be better in the end to make the CSVWriter functionality more transparent: i.e., provide some documentation. I think we will be needing to write several things to CSV in this project: production curves, demand curves, price curves etc. They will likely not fit the exact same format so it would be a waste of your time to build such specific functionality.

I think this is indeed more useful than putting more effort in a Merit 'dump' functionality as discussed before.

[ChaelKruip]

Part of #104

[antw]

Presently, CSVWriter is so basic that it may as well not exist. How happy are you to get "dirty" with Ruby when generating the CSVs? I also prefer your suggestion to just add documentation and examples describing how to get useful data out of the Merit::Order and participants. For example, this is how you might generate a CSV like the one ETEngine gives you:

columns = order.participants.producers.map do |producer|
  [ producer.key,
    producer.class,
    producer.output_capacity_per_unit,
    producer.number_of_units,
    producer.load_curve.to_a
  ].flatten
end.transpose

CSV.generate do |csv|
  columns.each { |column| csv << column }
end

The alternative is to simplify that by adding a new writer, but that's never going to be as flexible as using the core classes directly.

output = Dump.new(order.participants.producers) do
  dump       :key
  dump       :class
  dump       :output_capacity_per_unit
  dump_array :load_curve
end

output.to_csv

[ChaelKruip]

How happy are you to get "dirty" with Ruby when generating the CSVs?

Pretty happy :smile:

I tried to run your suggested code and ran into two issues:

• I think 'order' should be 'merit_order' (or 'mo') and 'participants.producers' should be replaced by 'producers'
• Executing the 'CSV.generate'-part I don't know how I can redirect the output to a file. Now it just writes everything to stdout.

[antw]

I think 'order' should be 'merit_order' (or 'mo')

"order" is just a variable; you can call it whatever you want. To use the (old) stub, a more complete example would look like:

# Start a "stub" console with: bundle exec rake console:stub

order = Merit.stub.calculate

columns = order.participants.producers.map do |producer|
  [ producer.key,
    producer.class,
    producer.output_capacity_per_unit,
    producer.number_of_units,
    producer.load_curve.to_a
  ].flatten
end.transpose

CSV.generate do |csv|
  columns.each { |column| csv << column }
end

For real-world use, we'd replace Merit.stub with a Merit::Order already set up with the project data.

and 'participants.producers' should be replaced by 'producers'

participants returns a ParticipantSet object containing all participants in the merit order. It has the following helper methods which are used internally by the library, but you are also free to use them in your own code:

• producers - All producers (volatiles + must_runs + dispatchables)
• volatiles
• must_runs
• dispatchables
• users - Things which create demand.
• transients - Producers which may be active sometimes, and inactive in others.
• always_on - Producers which are always running. Demand is assigned to these producers before the "transients".

We could certainly alias one or more of these on the Merit Order object if calling "participants.something" is too long.

Executing the 'CSV.generate'-part I don't know how I can redirect the output to a file. Now it just writes everything to stdout.

You have two choices:

1. Assign the CSV content to a variable, and write to the file in Ruby:

   # ...
   
   csv_content = CSV.generate do |csv|
    columns.each { |column| csv << column }
   end
   
   File.write("path/to/file.csv", csv_content)

2. Save the Ruby as a script (this won't work if you just paste the code into the console), and have the script output the CSV content to STDOUT:

   csv_content = CSV.generate do |csv|
    columns.each { |column| csv << column }
   end
   
   puts csv_content

   Run your script redirecting the output to a file

   ruby example.rb > path/to/file.csv

   Here is a complete script; save it to "example.rb" within your merit directory.

[ChaelKruip]

@antw thanks! That is all I need for now :+1:

[ChaelKruip]

@antw is it easy / possible to write price curves to files as well? I would like to read in the DE and NL price curves in my post-processing script and it is also the main deliverable to Berenschot.

[antw]

is it easy / possible to write price curves to files as well?

Which price curves are you looking for? A curve for the price of the whole region, or the final marginal/variable/fixed/... cost for each producer (accounting for demand)?

The first is already implemented as Merit::Order#price_curve.

mo = Merit.stub.calculate

csv_content = CSV.generate do |csv|
  mo.price_curve.each { |v| csv << [v] }
end

File.write('region-price-curve.csv', csv_content)

The second isn't currently supported, but should be trivial to implement.

[ChaelKruip]

The first is already implemented as Merit::Order#price_curve.

That one! Cool! :+1:

[ChaelKruip]

Closing.