search

openbudgets / platform

Tracking issues related to the working around the OpenBudgets.eu platform (WP4).
GNU General Public License v3.0
1 stars 0 forks source link

Select OBEU-datasets for upload to the OBEU-platform #16

Closed schmaluk closed 7 years ago

schmaluk commented 8 years ago

We need to exactly identify / select the OBEU-datasets from: https://github.com/openbudgets/datasets which are intended to get uploaded to our OBEU-prototype/OBEU-platform.

A complete list of OBEU-datasets from https://github.com/openbudgets/datasets :

1. Aragon:

2. Bonn:

3. Decin/2014/section-xi:

4. ESF-CZ/2007-2013:

5. ESIF/2014:

6. NSRF-GR:

7. Ustecky_region/2014/section-xi:

8. ecb-exchange-rates:

9. eu-budget/2014:

10. greek-municipalities

Please mark the OBEU-datasets which are relevant for the upload with a comment below. Only marked datasets will then get added to the triplestore & rdf filestorage on our OBEU-Server.

Users of the OBEU datasets which are invited to participate here are for example:

schmaluk commented 8 years ago

Dismissed Datasets from Děčín or Ústecký region since they were only used to illustrate the infeasibility of tracing EU funds according to @jindrichmynarz https://github.com/openbudgets/platform/issues/15

jindrichmynarz commented 8 years ago

I only used data from Děčín and Ústecký region as an illustration of what datasets may not be used any more. Otherwise, they are perfectly fine and can be used.

jindrichmynarz commented 8 years ago

I think we'll dismiss ecb-exchange-rates rates dataset, since we don't use daily exchange rates (at least I'm not aware of any use). Instead, we use yearly average exchange rate from http://eurostat.linked-statistics.org in the normalization pipeline fragment.

ChristianeEngels commented 8 years ago

"Aragon" and "greek-municipalities" should be uploaded to the platform. These datasets contain data for many years (and in the Greek case also for several regions) which is a good basis for data analysis.

schmaluk commented 8 years ago

To summarize: Only the following datasets should get uploaded to our Server via LinkedPipes so far:

will be uploaded to our Server so far.

Do you manage to upload those datasets to our Server until the 1st September @fathoni @larjohn @skarampatakis? So that we have datasets prepared on the server as input for datamining and for visualisation.

Here are some Information you might find useful for this task:

Should a named graph for each year in the dataset be created @jindrichmynarz @jakubklimek ? And what Name for the named graph should be Chosen here? Thanks

jindrichmynarz commented 8 years ago

Should a named graph for each year in the dataset be created?

Yes. I believe most of our pipelines already put datasets in specific named graphs. Regarding the construction of named graph IRIs, they should be the same as the IRI of the qb:DataSet instance in the dataset.

jakubklimek commented 8 years ago

Also, the metadata should be produced for each dataset as discussed in https://github.com/openbudgets/platform/issues/13

skarampatakis commented 8 years ago

Hi, I am trying to upload some datasets for test purposes on OBEU server. I am using LP for this, as default s-put commands seems not to be working from terminal. At first, it seems tha LP localFile component is not working. It throws an error as :

com.linkedpipes.etl.executor.api.v1.RdfException: Invalid property: http://plugins.linkedpipes.com/ontology/e-filesFromLocal#path

Local path is defined in component.

I switched to http-get component using the files on our github repo, to overcome this problem with success.

Now, Graph store protocol hits an error. When http://triple_store_staging:3030/fuseki/data is defined it throws uknown error. Editing to http://localhost:33030/fuseki/data gives a connection refused error.

The pipeline is on http://localhost:18080 LP instance under the name Dump upload test. Can anyone help?

jakubklimek commented 8 years ago

Seems to be some networking issue since I can access Fuseki on http://localhost:33030/fuseki/data @mlukasch any config changes?

jakubklimek commented 8 years ago

@skarampatakis btw the files from local path property should have been the path, not an IRI, i.e. /home/sotiris instead of file:///home/sotiris

skarampatakis commented 8 years ago

I have tried both with no success. It throws the same problem: Invalid property.

schmaluk commented 8 years ago

@skarampatakis No 18080 is not the right instance. 18080 & 28080 are originally for Christiane & Fathoni for their experiments. 8181 is the proper LP instance. But currently another pipeline is running. So you would need to wait. For the future I can provide more LP instances than just 1. So each LP developer would be able to use their own LP instance maybe. So you wont block each other. On the proper instance 8181 the url http://triple_store_staging:3030/fuseki/data should be reachable.

skarampatakis commented 8 years ago

Can you re-configure some of the other 2? It seems that on 8181 there is a long running pipeline.

skarampatakis commented 8 years ago

I have managed to run my pipeline on 8181 LP instance. However when I change HTTP get component to Files from local I get again same invalid property error. @jakubklimek is this an LP issue? I would like to be able to do so , because we have to upload all of our datasets. The other solution would be to get a list of all github IRIs and feed the HTTP Get List Component. How is this component get the list? Text holder? Also, can we configure Storage protocol component to store on seperate graphs using the dataset IRI as graph name? How can we achieve this?

schmaluk commented 8 years ago

Hello @skarampatakis , I have reconfigured the 18080 & 28080. You can use one of those now. The config is still the same as on 8181: http://triple_store_staging:3030/fuseki/data SSH-Server: sshserver, User & Password: root Will test also the other solution suggested by Jakub with the LinkedPipes instance on your local machine using ssh-connection to LP.

schmaluk commented 8 years ago

But since the Pipeline has stopped, I can restart the server now and rather setup some new Instances in OBEU-stack only for custom pipelines.

schmaluk commented 8 years ago

@skarampatakis There are now 5 new instances: 8282, 8383, 8484, 8585, 8686 of LP available on the Fhg-server as written in the mail.

skarampatakis commented 8 years ago

Hi all, I have used the LP instance on 8686 to bulk upload all Greek Municipalities Datasets. Can you check if everything is OK?

Some general codelists are missing, I will have them uploaded by today. Is there any specific requirement about in which named graph can we use? Or can we choose a generic one for test purposes such as http://data.openbudgets.eu/resource/codelist/greek-municipalities ?

jindrichmynarz commented 8 years ago

Regarding the named graphs, each dataset should be in a separate named graph, the IRI of which should be the same as the qb:DataSet instance's IRI.

skarampatakis commented 8 years ago

All datasets have seperate named graph which is the same with the qb:DataSet. But I was asking about the Code Lists.

jindrichmynarz commented 8 years ago

Sorry, I misread your point. The recommended treatment of code lists is similar: each code list should be in a separate named graph, the IRI of which should be the same as the skos:ConceptScheme instance's IRI.

skarampatakis commented 8 years ago

OK @jindrichmynarz, thank you for your immediate response.

pwalsh commented 7 years ago

@badmotor I'm closing this. It must be out of date, and any specific issues can be raised with distinct issues.