Dat Module Wishlist

[max-mapper]

This issue is to discuss ideas for dat modules that would be awesome to have. If you want to write a module leave a comment to notify others that you are working on it (preferably also leave a link to the repo).

I'll keep this post updated to link to all of the modules (until we have too many to fit here).

To learn more about dat modules check out the Importing data guide

If a module doesn't have a link w/ it it means nobody is working on it yet

Read about Import Methods to determine which type of module should be used for each case.

List

(not in any particular order)

In Progress/Done

• Google Drive - started
• Google Cloud Storage - started
• Postgres - done (experimental)
• Heroku Postgres - done (experimental)
• PouchDB - initial working version
• ElasticSearch - proof of concept
• MongoDB - mongodown (alpha), node-mongs (alpha)

  Up for grabs

• CouchDB
• CartoDB
• MySql
• MSSQLServer
• Redis
• Hadoop
• MongoDB
• Google BigQuery
• Amazon RedShift
• Amazon SimpleDB
• Amazon DynamoDB
• ESRI ArcGIS/ArcServer
• Microsoft Access
• Microsoft Azure Table Storage (maybe use this?)
• Microsoft Azure Blobs (details)
• IBM SQL Database for Bluemix
• Socrata Data Portals
• Elasticsearch
• FTP
• HTTP (open directory spider?)
• Filesystem (maybe w/ filewatcher support to update hashes?)
• Dropbox
• S3
• Spideroak
• Box
• IPFS
• Google Fusion Tables
• OpenStack Swift
• Ceph
• HDFS
• HDF5 (also see hdf5.node and h5py)
• Google Cloud SQL
• RethinkDB
• CKAN API, in particular through datahub
• firebase
• Neo4j
• NodeRED

[missinglink]

re: elasticsearch https://github.com/geopipes/elasticsearch-backend

[webmaven]

Some other storage possibilities (in no particular order, and I am not working on any of these):

• Spideroak
• Box
• IPFS
• Google Fusion Tables
• OpenStack Swift
• Ceph
• HDFS
• HDF5 (also see hdf5.node and h5py)
• Google Cloud SQL

[max-mapper]

@webmaven awesome list, thanks! I definitely have @jbenet on the hook to write an IPFS backend once it's ready

[webmaven]

You're very welcome, @maxogden.

[astro]

Here's one uploader for ElasticSearch based on the module by @missinglink: https://github.com/astro/dat-elasticsearch-upload

[thadguidry]

Support the CKAN API, in particular through http://datahub.io (part of CKAN - Open Knowledge Foundation) - Latest docs here: http://docs.ckan.org/en/latest/api/index.html

[calvinmetcalf]

pouch-dat is a connector for pouchdb/couchdb and dat

[knownasilya]

rethinkdb might be a good one to add as well

[wardi]

+1 on CKAN datastore API support, or even a CKAN extension that uses Dat instead of CKAN's datastore.

[RichardLitt]

Neo4j might be another possibility.

+1 for CKAN.

[coleww]

https://github.com/coleww/s3-blobs I make no promises, but i shall try!

[doowb]

Not sure if this fits here, but I just created vinyl-dat to read/write dat records in a gulp workflow. It's basically just a way to convert dat records into vinyl files. I think it still needs some work and possibly a better way to just let dat do conversions on contents (like choosing csv, json, etc...)

[doowb]

firebase - I'll update if I start on this.

[jcblw]

I think this should work for redis as a backend https://www.npmjs.org/package/redisdown

[finnp]

I am going to have a look at CKAN as an import/export target as well as a possible storage back-end. ckanDOWN: https://github.com/finnp/ckanDOWN

[pvgenuchten]

Would https://www.npmjs.org/package/gdal be a usefull lib for dat to connect to a huge number of formats (http://gdal.org/ogr_formats.html)?

[max-mapper]

@pvgenuchten that is awesome, thanks for linking it. I'm not sure what the best place to plug gdal into dat would be at the moment but I'll think about it. It's also great that the gdal module is using node-pre-gyp to make it easy to install cross-platform

[markhahnel]

We'd be interested in throwing http://figshare.com/ in the ring here, also have some devs who could help out :)

[calvinmetcalf]

So this is a pouchdb want originally but applies to dat.

Leveldown can be tricky to get working on all platforms and while pre-built ones ones are helpful there are always going to be weirder platform combinations.

It would be great to have a pure js leveldown backend (for node) that was not restricted to memory.

Being able to install dat or pouchdb with no c deps would be super useful.

Medea seems the best of what is out there but has a requirement that keys be kept in memory which combined with the node.js hard 2GB mem limit means that there are hard limits on the datasets you'd be able to put into it.

So really the thing that would be super helpful is a pure js nodejs db with spill to disk.

[missinglink]

Check out http://geogig.org/

"GeoGig is an open source tool that draws inspiration from Git, but adapts its core concepts to handle distributed versioning of geospatial data."

[kevinsimper]

Has many of the basic one not been covered? :) As seen here https://github.com/rvagg/node-levelup/wiki/Modules#storage

Or am I am wrong? I am a little confused because how is this dat related, isn't is more of Leveldb related, that is needs more adapters?

Or is there a good example on a plugin there is written?

[taterbase]

Just published dat-mongo to handle realtime updates from MongoDB with the listen hook.

[yoshuawuyts]

@calvinmetcalf If you're looking to build a pure JS levelDB this might be a good place to start -- https://github.com/rvagg/leveljs

[calvinmetcalf]

Ooo I like, thanks On Oct 23, 2014 2:04 PM, "Yoshua Wuyts" notifications@github.com wrote:

@calvinmetcalf https://github.com/calvinmetcalf If you're looking to build a pure JS levelDB this might be a good place to start -- https://github.com/rvagg/leveljs

— Reply to this email directly or view it on GitHub https://github.com/datproject/discussions/issues/5#issuecomment-60281261 .

[wikier]

any interest to RDF / Linked Data support? I can work on a dat-marmotta implementation for testing it.

[yueyingcui]

• Couchbase

[almereyda]

@wikier RDF / Marmotta here, please! We'll need it for http://transformap.co

[wikier]

so @almereyda, should we create an issue to organize the work, define the goals, etc?

[almereyda]

@wikier Yes, of course! We'll be squatting the CouchDB Meetup in Berlin next week, where I'm also inviting @finnp right now, to precise the requirements of the TransforMap data integration pipeline.

Please feel free to get started with your perspective, but I will only then be able to respond meaningfully.

[almereyda]

In the dawn of tomorrow, I'm stumbling over dat browser.

[shawnbot]

I can't believe nobody's mentioned SQLite. So I'll do it.

[balupton]

How about JSON, RDF, and Atom feed URLs? :-D

[almereyda]

Yeah, then I could transform Discourse' API into RSS :) Or we just plug dat on SocketHub. Or these feeds better live there.

On 7 June 2015 at 16:10, Benjamin Lupton notifications@github.com wrote:

How about JSON, RDF, and Atom feed URLs? :-D

— Reply to this email directly or view it on GitHub https://github.com/datproject/discussions/issues/5#issuecomment-109759205 .

[fabianvf]

Guide links in the issue are broken.

[onbjerg]

links to the guides are still broken

[webmaven]

@maxogden, perhaps edit the original comment on this issue to use [ ] [x] checkboxes instead of bullets?

[patcon]

+1 on CKAN