search

spine-tools / Spine-Toolbox

Spine Toolbox is an open source Python package to manage data, scenarios and workflows for modelling and simulation. You can have your local workflow, but work as a team through version control and SQL databases.
https://www.tools-for-energy-system-modelling.org/
GNU Lesser General Public License v3.0
73 stars 17 forks source link

Serialize and deserialize methods for project items #426

Closed spine-o-bot closed 3 years ago

spine-o-bot commented 3 years ago

In GitLab by @manuelma on Jan 15, 2020, 16:12

The idea here is to find a text representation for a ProjectItem instance, so these can travel in text files. From a given toolbox project, we generate a text file which contains all the information needed to recreate the project elsewhere. Connections (aka yellow arrows) are easy to translate into a JSON, but ProjectItems might be harder although we already do something related when saving the project file.

So in summary, the aim here is to implement ProjectItem.dump and ProjectItem.load.

Edit: I'm not sure, but this text representation may follow some standard? Does CWL apply here?

spine-o-bot commented 3 years ago

In GitLab by @manuelma on Jan 15, 2020, 16:12

changed the description

spine-o-bot commented 3 years ago

In GitLab by @manuelma on Jan 15, 2020, 16:14

changed the description

spine-o-bot commented 3 years ago

In GitLab by @jkiviluo on Jan 16, 2020, 10:24

Do @fabianoP or @ererkka have thoughts on this?

spine-o-bot commented 3 years ago

In GitLab by @PekkaSavolainen on Jan 16, 2020, 11:22

@jkiviluo this was discussed in Spine Engine telco yesterday.

@manuelma I may have missed something in the telco discussion but what is the use case for these ProjectItem.dump and ProjectItem.load functions other than make them do a CWL representation of the project item? This would be the next step for Issue #491. If there is no other use case then we should probably close this issue and work on this as part of Issue #491.

spine-o-bot commented 3 years ago

In GitLab by @manuelma on Jan 16, 2020, 11:40

I have trouble seeing the advantages of CWL to be honest. I don't know how well it plays with dagster. In dagster, the PipeLineDefinition constructor expects a list of solid definitions and a dictionary of 'dependencies'. That seems very clean to me.

In the current version of spine_engine, the solid definitions are extracted directly from the ProjectItem instance (execute method and some friends). So in my view, all we need is a way to dump and load project items.

So maybe dump and load could follow CWL somehow, but I don't really see why? If it's just to adopt an standard, well maybe that's sufficient reason and we can go with it.

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 31, 2020, 15:31

This is done somewhere. We are already able to port Spine toolbox projects through the project .json file.

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 31, 2020, 15:31

closed