Database of the sources documents (i.e. ENB publications) - Githubissues

rufuspollock / climate-negotiations

Information on the UNFCC climate negotiations using the Earth Negotiations Bulletin from the IISD

https://rufuspollock.github.io/climate-negotiations/

3 stars 0 forks source link

Database of the sources documents (i.e. ENB publications) #1

Closed rufuspollock closed 8 years ago

rufuspollock commented 8 years ago

Store at: data/documents.csv

What is wanted:

id
source url
date
event - event id from event table #14

Do we already have this?

If not we can sort of get it from the txt files we have.

/cc @tommv

tommv commented 8 years ago

Yes we do have them:

the name of the txt file is directly derived from the URL of the corresponding report (e.g. 1201000e.txt refers to http://www.iisd.ca/vol12/1201000e.html)
This table (https://github.com/medialab/climateDebateExplorer/blob/master/ENB-data/metadata_overview/event_bulletinurl.csv) connect the URLs of the reports with the ID of the events
and this other table (https://github.com/medialab/climateDebateExplorer/blob/master/ENB-data/metadata_overview/events_metadata.csv) connects the ID events with a bunch of meta-data on the events

rufuspollock commented 8 years ago

@pauloborges for this and the #14 (DB of events) we may just want to rescrape http://www.iisd.ca/enb/vol12/ as we get much more info e.g. issue no.

pauloborges commented 8 years ago

Just pushed this change. Check the script and both databases.

rufuspollock commented 8 years ago

@pauloborges looks good. Some quick comments:

documents.csv
- date column value is sometimes summary - that can't be right ...
- I suggest document id is the file name with extension stripped - that way it corresponds to documents with have stored
events.csv
- suggest renaming name to title in column names
- could we have title in the last column so stuff is more readable
can we get a datapackage.json with those CSVs in it and description for columns. BTW https://github.com/okfn/dpm is your friend here ;-)

pauloborges commented 8 years ago

@rgrp,

Every event has one issue that is a summary of the event. It has no associated date, as you can see in the original page. What should I put as its value?

pauloborges commented 8 years ago

I've done the other suggested modifications, including adding a datapackage.json file. You can check it using the Data Package Viewer.

Please check the columns descriptions since my english isn't very good.

rufuspollock commented 8 years ago

@pauloborges i would put the date as the last day of the event.

pauloborges commented 8 years ago

@rgrp, ok! I'll change this as soon as possible.

pauloborges commented 8 years ago

@rgrp, done!

rufuspollock commented 8 years ago

FIXED.