search

ReeemProject / reeem_db

Code of the reeem_db
GNU Affero General Public License v3.0
6 stars 4 forks source link

Database Adapter - TIMES PanEU #4

Open Ludee opened 6 years ago

Ludee commented 6 years ago

modeller: @pkork

Ludee commented 6 years ago

Recent problems:

Ludee commented 6 years ago
Ludee commented 6 years ago
Ludee commented 6 years ago
table pathway version count
Input Base DataV1 77392
Input Base DataV2 81424
Input HighRES DataV1 78176
Input Pilot DataV1 23416
Input StorageInnov DataV1 77392
Output Base DataV1 420672
Output Base DataV2 437424
Output Base(withRen.Target) DataV1 421984
Output HighRES DataV1 421984
Output Pilot DataV1 421992
Output Pilot2 DataV1 421976
Output StorageInnov DataV1 421984
Ludee commented 6 years ago

I found some inconsistencies in the numbering of the indicators. Every indicator should have a unique nid.

I think this is due to a bug in the database adapter and appears on the Base pathway of the output.

4lm commented 5 years ago

Hi @pkork,

the sources info in the metadata of the input model data of TIMES PanEU is incomplete, please follow this instruction to provide the infos.

pkork commented 5 years ago

In the excel, i provided I see that they are complete. The sources are either IER research or WP2 roadmaps. Do you refer to the information in the column P actually or you are asking additional work to do? I'm sorry it is not clear for me:(

Ludee commented 5 years ago

In the TIMES input data, there is a column named "sources" you are refering to. @4lm is refering to the "metadata string" for the entire table. The metadata uses JSON syntax (key-value). Have a look at this example.

The link directs to the corresponding lines 62-64 for the input table:

"sources": [ {"name": "", "description": "", "url": "", "license": "", "copyright": ""}, {"name": "", "description": "", "url": "", "license": "", "copyright": ""} ],

You need to fill out these 5 keys! Use "none" if na. Adding "IER research" as your source is possible, but IMHO this is scientific BS. But I know it quite difficult to track every value and source.

4lm commented 5 years ago

Hi @pkork, hi @Ludee,

I wanted to get an understanding of all sources for the input data, so that I can assist adding them to the metadata JSON string. I had a look at the REEEM database and in the table "reeem_times_paneu_input" I aggregated all the sources for all pathways, frameworks and data versions using this SQL query:

SELECT source, COUNT(*) AS amount, ROUND((COUNT(*) * 100.0 / (SELECT COUNT(*) FROM model_draft.reeem_times_paneu_input)), 2) AS percentage 
FROM model_draft.reeem_times_paneu_input
GROUP BY source;

As a result I get 2 sources:

source amount percentage
NULL 32586 2.30
WP2 Roadmap F1_TI1_P1 30240 2.13
IER research 1354142 95.57

Please note: The NULL values are from the Pilot pathway and the EU28 aggregations.

@pkork, for this two sources we now need a closer description, source-url, license and copyright terms. Please follow the general instruction to do so: https://github.com/ReeemProject/reeem_db/issues/55#issuecomment-492234763

4lm commented 5 years ago

Hi @pkork, any news concerning the source infos for the metadata?

4lm commented 5 years ago

Hi @pkork, also the license info in the metadata for the input data is stil set on tba: https://github.com/ReeemProject/reeem_db/blob/master/database_setup/reeem_db_setup_times_paneu.sql#L65-L71

4lm commented 5 years ago

Hi @pkork,

me again :)

There is a file in the Sharepoint Model_Data folder, named:

2019-05-06_All Pathways_TIMESPanEU_Input.xlsx

Two things:

Are you OK with this changes?

@Ludee, what do you think? Is this viable (especially my second "DataV0" suggestion)?

Edit: I also would set the missing framework value to "FrameworkV0"

EditEdit: @Ludee, I just see, that we used in another model "PathwayNA", so I assume "NA" stands for not available, we also could use this here. So, "DataNA" and "FrameworkNA" instead of "DataV0" and "FrameworkV0" - what do you think? I find both solutions OK, but we already used "NA" in such a case, so I would stick with this solution ...

pkork commented 5 years ago

I’m ok with the changes and I will come back to other points within next week. Regards, Pinar From: Alexis Michaltsis [mailto:notifications@github.com] Sent: Wednesday, May 29, 2019 5:04 PM To: ReeemProject/reeem_db reeem_db@noreply.github.com Cc: Pinar Korkmaz Pinar.Korkmaz@ier.uni-stuttgart.de; Mention mention@noreply.github.com Subject: Re: [ReeemProject/reeem_db] Database Adapter - TIMES PanEU (#4)

Hi @pkorkhttps://github.com/pkork,

me again :)

There is a file in the Sharepoint Model_Data folder, named:

2019-05-06_All Pathways_TIMESPanEU_Input.xlsx

Two things:

Are you OK with this changes?

@Ludeehttps://github.com/Ludee, what do you think? Is this viable (especially my second "DataV0" suggestion)?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ReeemProject/reeem_db/issues/4?email_source=notifications&email_token=AIBB3WM6FY72VYRYJSUNYYDPX2LOZA5CNFSM4EB6INL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWPUDNQ#issuecomment-496976310, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AIBB3WKK3CECCXIQ6DCK4WLPX2LOZANCNFSM4EB6INLQ.

4lm commented 5 years ago

IMPORTANT: Request for OEP-Publishing

Hi @JuliaIER and @pkork,

you licensed reeem_times_paneu_output as CC-BY-4.0 and therefore suitable for publishing on the Open Energy Platform (OEP). We now pre-staged the data on OEP and are ready to hit the button :) Nevertheless we want to ask for your final OK!

If you want to have a look at the data, you find it in the REEEM DB in model_draft.reeem_times_paneu_output

Thanks in advance!

4lm commented 5 years ago

Hi @JuliaIER and @pkork,

a little reminder, we still need the input sources and license info of TIMES for the input metadata.

Please provide us now with the metadata of your data, just post it in a comment in this issue thread!

SOURCES: "REEEM Times PanEU Input"

"sources": [
        {"name": "none", "description": "none", "url": "none", "license": "none", "copyright": "none"},
        {"name": "none", "description": "none", "url": "none", "license": "none", "copyright": "none"} ],

LICENSE: "REEEM Times PanEU Input"

    "license":
        {"id": "none",
        "name": "none",
        "version": "none",
        "url": "none",
        "instruction": "none",
        "copyright": "none"},

Thanks!

CC: @Ludee

4lm commented 5 years ago

IMPORTANT: Request for OEP-Publishing

Hi @JuliaIER and @pkork,

you licensed reeem_times_paneu_output as CC-BY-4.0 and therefore suitable for publishing on the Open Energy Platform (OEP). We now pre-staged the data on OEP and are ready to hit the button :) Nevertheless we want to ask for your final OK!

If you want to have a look at the data, you find it in the REEEM DB in model_draft.reeem_times_paneu_output

Thanks in advance!

Hi @JuliaIER and @pkork, what about this request? What is the status here?

CC: @Ludee