search

humanitiesplusdesign / palladio

An application that brings humanities research methods to data visualization.
BSD 3-Clause "New" or "Revised" License
172 stars 32 forks source link

Database connection #181

Closed omoratto closed 7 years ago

omoratto commented 7 years ago

I just found palladio and found it very interesting. I was wondering, there is a way to connect palladio to a database (i.e., oracle, sql, etc.) in order to ingest masive data?

Thanks

esjewett commented 7 years ago

Hi - welcome! Palladio will not work well with large datasets, as it runs completely in the browser. You can ingest data from URLs in tabular format, or from SPARQL endpoints, so if you want to programmatically provide tabular data via a URL (CSV would work best), then you could use Palladio on dynamic data. But you'll want to keep the data volume under ~10,000 records.

omoratto commented 7 years ago

Thank you. The idea is to bring data dynamiclly, connected to a database, that the analyst can "search" or "query" the database directly into Palladio. It can be done? there is any example available? Thanks

esjewett commented 7 years ago

Well, Palladio doesn't push down any of its query/filter information. It just loads in flat files. So if you can expose the data from your database as a flat file at a URL, just put that URL into the data field of Palladio and it will load it. Would that work for you?

omoratto commented 7 years ago

I would need to pull data dynamicly in order to ingest it into Palladio. It can be done? If so, Do you have any examples?

esjewett commented 7 years ago

No examples of dynamic data that I'm aware of. You would need to set up a web server/application to do the database query you need and provide the output as a flat file format (CSV). Palladio can ingest that URL. There is a sample file at this URL that you can test out in Palladio to see more or less how it works: http://hdlab.stanford.edu/palladio-app/sample%20data/Letters.txt

Here is a short, related tutorial: http://hdlab.stanford.edu/palladio/tutorials/publicdata/

These are for static files, but the same idea would work for dynamically generated files. I wish there were simpler tools to make this easier for you, but unfortunately I am not aware of any at this time.