search

spine-tools / Spine-Toolbox

Spine Toolbox is an open source Python package to manage data, scenarios and workflows for modelling and simulation. You can have your local workflow, but work as a team through version control and SQL databases.
https://www.tools-for-energy-system-modelling.org/
GNU Lesser General Public License v3.0
69 stars 17 forks source link

Combiner project item - [merged] #1174

Closed spine-o-bot closed 3 years ago

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 13, 2020, 01:14

_Merges recombinator_projectitem -> dev

The idea is simple: connect any number of Data Stores as inputs and any other number as outputs. When executed, all the data from the inputs is merged into each of the outputs.

Selective merge (i.e., select which items to merge from the input dbs) should be done in another Project Item (e.g., Filter).

Re #518, and #570 (theoretically, the stack of changes could simply be some dbs holding the differences with respect to the base case, so we can then merge all of them with this item into the definitive version).

To test, just connect Data Stores in and out and press execute. The cancel on error setting should also be remembered by the project, and undoable.

The name 'Combiner' is also open for discussion. Other nice options are Merger, Mixer, even Blender. The problem is they are all equally good.

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 13, 2020, 01:15

changed the description

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 13, 2020, 01:16

changed the description

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 13, 2020, 03:41

added 2 commits

Compare with previous version

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 13, 2020, 05:02

added 2 commits

Compare with previous version

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 13, 2020, 05:05

changed the description

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 13, 2020, 13:28

added 5 commits

Compare with previous version

spine-o-bot commented 3 years ago

In GitLab by @soininen on May 14, 2020, 09:35

Unfortunately I could not test this as the branch seems to be broken ATM.

In any case, I have two general concerns:

First, I am unsure how to proceed with the 'full fledged' version. The thing is that 'selective export' might not be enough. Users may start asking for entity renaming on the same go. And then something more. What I am concerned here is that we might get one big Recombinator item which is actually a Swiss army knife of all database operations. Perhaps we shoud consider having a number of project items each performing a single basic task. Could that be more manageable?

Second, while I know the motivation behind this item is database-to-database import-export, I wonder if this is the way we should do it. If we have multiple manipulators, say a renamer and a filter, do the users need to create an intermittent database and a corresponding Data Store between each of these project items? A single Swiss army knife manipulator would solve the issue by handling all the data in memory within the project item...

One solution for the second issue we discussed here at VTT (but never got to implement it) was something like this:

  1. a manipulator, say a renamer, stores its settings to a file/database/whatever in a format recognized by spinedb_api
  2. when a DatabaseMapping is constructed in the successor item with an URL pointing to the settings file the DatabaseMapping goes to a special mode where it translates/renames/filters/manipulates the queries according to the settings

In this scheme one does not need to itermittent databases.

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 14, 2020, 14:19

added 8 commits

Compare with previous version

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 14, 2020, 14:24

added 1 commit

Compare with previous version

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 14, 2020, 14:28

I have now fixed the branch, thanks for the heads up!

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 14, 2020, 14:37

If I hear you well, the first concern is that you would prefer not to do the 'filtering' (aka selective export) in the Recombinator, but rather in another project item (say, Filter) designed especially for that purpose. I can get behind that.

The second concern is about chaining DB manipulations and is indeed very interesting. I think we can make it so all "Manipulators" pass data in memory dbs or in a lighter format to other Manipulators; and only when the workflow reaches a Data Store, the data is written into the target database. And the data that is written is the result of all accumulated 'manipulations'.

So the second concern is not about the Recombinator per se, but about how to handle the case where the Recombinator will interact with other Manipulators?

spine-o-bot commented 3 years ago

In GitLab by @soininen on May 14, 2020, 15:54

Yes, basically none of the concerns is directly about the Recombinator here but more general about which way we want to go from here.

You got it right that in the chained manipulator scenario each manipulator would write to a Spine database if they get a database reference from a successor Data Store.

As a practical concern the manipulators would probably need to store their manipulation settings on disk and pass a URL (or a stack of URLs including settings from previous manipulators!) to successors.

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 14, 2020, 16:16

You're right about that last practical concern! I didn't think about that and indeed might be a lot of fun to get it right.

But I propose we limit the scope of this MR to the scenario where the Recombinator is the only Manipulator around. As other Manipulators get added (and I feel the next could be the Filter) we can solve those issues. What do you think?

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 14, 2020, 16:46

Also, the name Recombinator was trying to capture Slicing and Combining in a single word, but if we'll do the slicing in another manipulator I think we can rename the Recombinator to just Combiner (or Mixer).

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 14, 2020, 16:54

added 1 commit

Compare with previous version

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 15, 2020, 01:43

added 1 commit

Compare with previous version

spine-o-bot commented 3 years ago

In GitLab by @soininen on May 15, 2020, 08:16

I agree, we should keep this MR as is. It was not my intention to suggest we start adding this stuff right here. I just happened to hijack this MR for my propaganda purposes :)

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 15, 2020, 12:42

changed title from {-Recombinato-}r project item to {+Combine+}r project item

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 15, 2020, 12:42

changed the description

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 15, 2020, 15:27

Your comments were really on point, we need to discuss them in the next dev meeting.

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 15, 2020, 15:37

added 15 commits

Compare with previous version

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 15, 2020, 17:44

added 2 commits

Compare with previous version

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 15, 2020, 17:53

added 1 commit

Compare with previous version

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 15, 2020, 18:16

added 3 commits

Compare with previous version

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 17, 2020, 22:38

added 15 commits

Compare with previous version

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 17, 2020, 22:42

added 2 commits

Compare with previous version

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 17, 2020, 22:55

added 1 commit

Compare with previous version

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 20, 2020, 15:00

Any volunteers to test? @PekkaSavolainen @Antti-L @ererkka @pvper @nnhjy ?

spine-o-bot commented 3 years ago

In GitLab by @nnhjy on May 20, 2020, 17:42

I've just had a try. The combinator combined and imported two DBs into one DB.

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 22, 2020, 20:51

Thanks very much @nnhjy !

So I guess it works... the question is whether or not we want it? Any comments @soininen @PekkaSavolainen @ererkka ?

spine-o-bot commented 3 years ago

In GitLab by @ererkka on May 25, 2020, 10:43

I’m pretty sure this kid of functionality will come handy. Is a new datastore (database file) created or is the new item reading from the ones combined? Can more than two stores be combined?

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 25, 2020, 14:38

You can combine any number of stores and they are combined in memory (into a python dictionary, that can be splatted as keyword args for spinedb_api.import_data).

Yes, I also think this is not something we'll use right now but it's ok to merge it now. If you don't mind, I will go ahead and do it. Unless @PekkaSavolainen and @Antti-L have any last words?

spine-o-bot commented 3 years ago

In GitLab by @PekkaSavolainen on May 25, 2020, 15:14

Are there any other changes in addition to a new project item in this MR?

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 25, 2020, 15:21

No, that's basically it. I was thinking I'll fix merge conflicts, make sure the tests pass and then merge into dev.

spine-o-bot commented 3 years ago

In GitLab by @PekkaSavolainen on May 25, 2020, 15:22

Cool, go ahead.

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 25, 2020, 17:01

added 58 commits

Compare with previous version

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 25, 2020, 17:21

added 1 commit

Compare with previous version

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 25, 2020, 17:21

merged

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 25, 2020, 17:21

mentioned in commit 489fbb58022b949e6020348feac5166116de1c31

spine-o-bot commented 3 years ago

In GitLab by @manuelma on May 25, 2020, 17:31

Done, thanks very much.

spine-o-bot commented 3 years ago

In GitLab by @DillonJ on Sep 7, 2020, 15:17

A high-level point here - there is mention of the idea of a base case in the original issue description. Much of the alternatives and scenarios implementation is achieving the same thing. How does this feature relate to scenarios and alternatives and can they be used nicely together?