Standard.csv merge service

[ColmDC]

Is your feature request related to a problem? Please describe. Merging data about co-ops from different sources can be done in many ways. It has been made easy by the efforts we have put into considering standards for the data we publish, however we have not written any data merging functionality for the data factory and we publish only one map demonstate co-op data set merging, and there, the merging was carried out within Airtable.

Describe the solution you'd like A function which can be easily called from the data factory that can take the following files:

• a primary standards csv file;
• a secondary standards csv file;
• a list which matches IDs from the primary file with entries in the secondary;
• a list of the fields in the secondary, not present in the first that it we wish to add;

and merges the 2 csv files where the resultant csv file has the following characteristics:

• it includes data for all the co-ops present in the primary data set, and no other;
• it include all the fields in the primary source and the specified new fields in the secondary;
• using the list of matches, it takes the values in the additions fields from matched co-ops in the secondary file and populating the fields in the primary co-op entry;

Describe alternatives you've considered We have considered doing merging on a co-op by co-op basis as the user selects a co-op, with a Sparql enabled micro service but we believe this may not be very responsive.

Additional context For ease of incorporation into the Data Factory, the function should be written in Ruby, but there are other factors to consider, so we should consider the options. There are many other types of merge we can imagine needing in the near future, so it may be worth considering functionality a little more generic than strictly required for this type of merge, if the cost was not too high.

[ColmDC]

So at dev.data.solidarityeconomy.coop/coops-uk/standard.csv find the csv published for co-ops uk data, but there is nothing at https://dev.data.solidarityeconomy.coop/dotcoop/standard.csv

Are the paths different for some reason? @wu-lee

[ColmDC]

Where can I find the standard.csv for current dotcoop data? @wu-lee

[wu-lee]

It should be there now - the SeOpenData gem was an older version which didn't publish this file, I updated it.

[ColmDC]

Dropping this out of the two Epics relating to the Turtle island confereces, as it is a proposed merge solution that is unlikley to be used now.

[ColmDC]

Closing this as obsolete @wu-lee