Use metadata to recommend visualization types

[pitsios-s]

We will provide data types and units for labels

[fabiankirstein]

I added myself to keep track of it.

[fabiankirstein]

I created an example, so you can start to integrate some logic in the visualizer. https://github.com/aegisbigdata/aegis-ontology/blob/master/examples/dataset.json This is how the Linked Data payload will look like. (I will deploy the service this week). I guess, for now you can just put it in a string. It is based on this CSV: hdfs:///Projects/AutomotiveDemonstrator/Demo2_SafeDrivingIndicator/Events/Driver1_Vehicle6_20170224_Trip_011__withoutnan.csv It reads the following. Each file is a disrtribution and the accessURL links to the file:

        "@id": "http://www.aegis-bigdata.eu/set/distribution/1234",
        "@type": [
          "aegis:TabularDistribution",
          "dcat:Distribution"
        ],
        "aegis:hasField": [
          {
            "@id": "_:ub1300bL41C30"
          },
          {
            "@id": "_:ub1300bL51C30"
          },
          {
            "@id": "_:ub1300bL36C30"
          },
          {
            "@id": "_:ub1300bL46C30"
          }
        ],
        "dcat:accessURL": {
          "@id": "hdfs:///Projects/AutomotiveDemonstrator/Demo2_SafeDrivingIndicator/Events/Driver1_Vehicle6_20170224_Trip_011__withoutnan.csv"
        },
        "dcterms:description": "This is an example distribution",
        "dcterms:format": "CSV",
        "dcterms:title": "Example Distribution",
        "hops:fileId": "3804679"
      },

For each column of the table you find a resource:

 {
            "@id": "_:ub1300bL46C30"
  }

And each resource gives you information about the column:

      {
        "@id": "_:ub1300bL36C30",
        "aegis:description": "The latitude of the entry",
        "aegis:name": "latitude",
        "aegis:number": "1",
        "aegis:type": {
          "@id": "aegis:Latitude"
        }

This should help to pre-populate some of the fields for the Map visualization. What do you think? Can you play around with it? This is JSON-LD. So you need some RDF Library, I'm not sure if you parse it in Python or JavaScript. For Python I recommend: https://rdflib.readthedocs.io/en/stable/ and for JavaScript we will use: https://www.npmjs.com/package/jsonld

[pitsios-s]

Thanks @FabiApfelkern , I will have a look at it.

[fabiankirstein]

Hi @pitsios-s,

a test dataset is also now available via an API: http://aegis-metadata.fokus.fraunhofer.de/datasets/test-dataset?useNormalizedID=true This call gives you the dataset level. More important for you is the file aka distribution level: http://aegis-metadata.fokus.fraunhofer.de/distributions/0d5ecb70-6b6a-4ca5-a7fd-3c40f992aaf9

[fabiankirstein]

I will refine it further soon.

[fabiankirstein]

Hi @pitsios-s You can now get data on the file level and the service is deployed on the testbed: https://bbc6.sics.se:8181/hopsworks-api/aegis-metadata/distributions/abcd?useIdentifier=true What is you current state? Thanks!

[pitsios-s]

Hello Fabian, thanks, that's great news. I will start working on it the soonest possible. I have two questions on this.

• Since, from what I understand the metadata service is included in the hopsworks API, do I need to get a JWT before making a call to it?
• What is the dataset identifier that I should provide (/abcd in your example)? Is it just the name of the file, or the whole hdfs url?

[fabiankirstein]

It has not auth yet. It is just open. The identifier will be the Hopsworks file ID. To avoid misunderstandings: It is only available when you add metadata. We will have the frontend for this early next week.

[pitsios-s]

Awesome. I think that I have what I need for now, so I will start working on the integration. If something comes up during the process, I will contact you.

[fabiankirstein]

Cool! :)

[pitsios-s]

This issue was properly addressed in merge request #81