DAQ smart devices: API to automate publishing of live data collected on my research experiments

[aeonSolutions]

Overview of the Feature Request integrate it into my own DAQ smart devices, to automate publishing of live data collected from my research experiments My open project related to this request can be found here

What kind of user is the feature intended for? (Example users roles: API User, Curator, Depositor, Guest, Superuser, Sysadmin) API, Depositor

What inspired the request? I'm part of open source community since ...forever. And currently, i have here on Github more than 20 PCB designs for smart devices.

What existing behavior do you want changed?

Any brand new behavior do you want to add to Dataverse? What i liked more above all others was the fact of providing a unique number fingerprint UNF plus a DOI web address link. However is not enough for me as i work a lot with LIVE data acquisition. Is there any API for me to use, so I can integrate it into my own DAQ smart devices, to automate publishing of live data collected from my research experiments ?

Currently only file uploads are allowed.

It would be great to upload:

• one line of CSV data records to append on a existing CSV file (no database, standard file).
• Alternatively upload a data record to a SQLite database file hosted on the repository.

This request was initially published here: https://www.linkedin.com/posts/migueltomas_harvard-dataverse-platform-activity-6971002717045407745--Dn6?utm_source=share&utm_medium=member_desktop

Any related open or closed issues to this feature request? no

final note: i'm available to provide my smart DAQ devices programmed with dataverse API for live data uploading to a dataverse repository

[pdurbin]

@aeonSolutions hi! I saw your note the other day that you'd like to contribute to the code! Were you thinking about this issue? Or some other issue?

Definitely in the context of the "replace file" feature or the idea of "streaming" data (e.g. #4530) we've discussed how it might be nice to append a row or two to a CSV file rather than storing full copies.

I believe others have suggested that Dataverse could host databases, but this hasn't happened, whether isn't MySQL, SQLite, etc.

Hopefully Dataverse already has enough APIs to integrate publishing of live data from your DAQ smart devices. I guess I supposed I'd suggest using the "replace file" functionality until you're ready for a new version (nightly? weekly?) and then publish.

I was on vacation when you popped in https://chat.dataverse.org before but I'm back if you want to chat. Thanks!

[aeonSolutions]

@pdurbin I'm open to contributing to any issue or other, for instance, new functionalities requests/roadmap.

I agree. Experimental data is not all that compatible with traditional/conventional cloud database architecture. Say MySQL.. Hence my suggestion of storing on single individual database files. Say a SQLite DB file. Much more portable and IPFS compatible.

adding to the API the ability to insert "a row" of live data to a CSV file sounds awesome to me, if not because most researchers like and use MS excel and CSV files are nothing new to them.

Another awesome feature to have on the API is the ability to MS Excel's Power query do experimental data download/retrieval without actually downloading the dataset file itself. To take full advantage of power query ability to do LIVE collaboration within MS Excel.

In regards to my own prototype smart DAQ device, I will make it available as soon I have a minimum working firmware doing basic experimental data upload to a demo repository,. almost there. Anyone can follow progress on my Twitter profile @AeonLabss or on LinkedIn here: https://www.linkedin.com/in/migueltomas/recent-activity/

[pdurbin]

I'm open to contributing to any issue or other, for instance, new functionalities requests/roadmap.

@aeonSolutions fantastic! To get started, how do you feel about working on a small Hacktoberfest issue to get a feel for out process? Please see my post about Hacktoberfest here for a list of issues or you can suggest a different issue: https://groups.google.com/g/dataverse-community/c/X8Q0cz0ERQ8/m/k5NKkuIUBgAJ Thanks!

[aeonSolutions]

Already opened an account