building block regarding databases

tilburgsciencehub / website

Learn to work more efficiently on empirical research projects.

https://tilburgsciencehub.com

38 stars 46 forks source link

building block regarding databases #1088

Closed MalihehMahlouji closed 3 months ago

MalihehMahlouji commented 4 months ago

As suggested by @hannesdatta:

It may be nice to craft the structure of a building block for getting a database in Tilburg, choosing the right setup, etc, and writing data to it in R.

lachlandeer commented 4 months ago

Hi @MalihehMahlouji

Can you elaborate on this some more? 4-5 bullet points on content that you want to see put in a building block would help us know a bit more precisely what you want developed and who might be best to work on this issue

Thanks!

MalihehMahlouji commented 4 months ago

Hi @lachlandeer sure.

Motivation: As a data steward at TiSEM, I am involved in big data collection projects. To reliably store big data, the best practice is to use a database instead of a csv. However, I often face with resistance from researchers towards implementing a solution involving a database due to various reasons. This building block hopefully can address some barriers regarding use of databases.

My proposal for the topics are as follows:

[x] csv vs database, compare the pros and cons
[ ] locally hosted databases vs. remotely hosted databases
[ ] overview of open access databases and when to use each
[ ] ? TiU databases and how to access them: as far as I know, there is not a stablished procedure yet. But I just asked someone from DRS to confirm this.
[ ] Tutorials on how to work with databases (SQL etc) and its integration with python/R.

hannesdatta commented 4 months ago

Hi @MalihehMahlouji - I like it. Can we please add a point about locally hosted databases vs. remotely hosted databases? I feel many researchers could benefit from SQLite databases that they just run on their own computers.

I, too, am very very hesitant about using databases that run live. First of all, data can't be archived for long-term. You always need a running instance. Second, it costs a lot of money to keep databases running. Third, for many applications, simply dumping to JSON or CSV is sufficient and doesn't require all the engineering to learn SQL (at least for many colleagues).

I'd say: great article, go ahead. The focus on (a) when it makes sense to go local, and (b) when it makes sense to go to the cloud/hosted at TiU, plus (c) tutorials/snippets for R & python sounds great!