Develop data parser - Githubissues

IndEcol / IE_data_commons

Code and documentation for a commons of structured industrial ecology data

MIT License

22 stars 2 forks source link

Develop data parser #1

Closed nheeren closed 6 years ago

nheeren commented 6 years ago

@nheeren will develop a parser for the existing database Excel files. They all have a similar structure (list or table). It has the following requirements:

consistency check 1: Is the data structure consistent with the data inventory?
consistency check 2: Do the classifications exist in the specified classification set?

nheeren commented 6 years ago

@stefanpauliuk We need to specify this a bit more. Could you please provide some indications for the following questions?

What xls files are you referring to? The RECC database files from Dropbox? Blatt Values?
What /where are the classifications to check against?
What is the exact interface? I.e. what data format should the parser deliver? Pandas dataframes? Upload directly onto MySQL?

stefanpauliuk commented 6 years ago

There are two xls files, one with the dataset descriptions and one with the structured datasets. The setup up is similar to the RECC datafiles but different, as more metadata are supplied here. The classifications are in a sheet similar to the RECC project classification. Will share all files via Dropbox. Parser will upload directly to mySQL review database. From there we download data via standard user interface. If downloaded data are the same as raw data, the dataset is moved from review database to 'real' database.

nheeren commented 6 years ago

In progress: https://github.com/IndEcol/IEDC_tools