Migrate data from data blocks to a data store

[mferrare]

1. Identify data.
2. Write pandas module to process
3. Migrate data to Google Sheets
4. Pandas module connects to google sheets

Data YAML files eg: list of services.

[mferrare]

I wrote a post on Slack about this. I've copied my post and one of the replies to the thread to comments here (as they expire on Slack after 90 days).

In short, I'm coming down to two options for a data store:

1. Use Airtable as suggested and implemented in the Slack discussion); or
2. Find a way to set up Django and interface with it.

If Airtable has enough of a free tier to support our clients then it may be the preferred option

[mferrare]

My Slack post

Hello everyone,

I am looking for a simple but functional way to manage data as part of a Docassemble application.

I use Docassemble for teaching in an undergrad law context. Our students are grouped into project teams and are paired up with a not-for-profit (NFP) 'client' who commissions them to write an app.

Many of the NFP clients come to us with 'service directory'-type apps. These apps usually generate a list of services (eg: legal clinics, family law services, domestic violence shelters etc) based on criteria such as age, gender and so-on.

Part of writing these apps involves storing a database of services in the app. We usually do this using a data block as this is the easiest to teach the students. However, updating a data block is nigh on impossible for our clients, who frequently want to commission these apps into their organisations. It's not practical for our clients to update data blocks by hand. It would be much easier if they had some guided way of managing their data.

I've been looking at a few things but none really 'float my boat' just yet. I wanted to put them out there along with some ideas and see what others thought or how others have tackled this problem.

Spreadsheets

The quickest and nastiest way to get data into an app would be by way of a CSV or XLS import. It wouldn't be too involved to write a library of sorts that students to paste into their codebase. Similarly, it wouldn't be too much of a bigger step to interface with Google Sheets or (eek!) Microsoft Graph.

The problem with this avenue though is twofold:

1. Data validation is non-existent in CSV files. With XLS files I could write validation rules into the XLS but that essentially has me supporting another app.
2. Data is limited to two dimensions. For example, a list of DV services may have a 'Location' column which would contain a list of locations (ie: a third dimension). Spreadsheets don't do so well in this context, especially as dimensions increase.

My own database(s)

I'm imagining a MySQL or PostgreSQL server which I run independently of DA and create routines so students can connect their apps to their data in the database.

This solves the data validation and multidimiensional issues but introduces new issues, the first of which comes to mind is that I'd have to write my own CRUD routines so students (and then the clients) can maintain their own data. I'm not and don't really want to be a web developer. There's the whole issue of supporting an extra server, managing access etc.

Django (or someting similar)

I've only started playing with Django but what I like is how the database schema is abstracted into Python and how it provides, by way of its admin interface, the data management tool of my dreams.

Django's appealing because the web development goes away, it manages authentication, and Python! I could hook up the abovementioned PostgreSQL database to Django and so provide the DA libraries for accessing data and have Django do the CRUD work.

If it's possible I could also interface to the data via Django itself (don't know that much about it yet)

Anyway, these are the things I have in mind. I'm looking for an easy way for clients to update their data in a service-directory type of application without having to write a data management system myself or having to teach our clients Docassemble. I'd appreciate any comments anyone might have on this.

Thanks

Mark

[mferrare]

The reply that uses Airtable

The links contain a neat app that finds services by location using list and a map

---

+1 to Airtable; I've seen other projects similar to this that use Airtable to great effect.

The poster from the Suffolk student is actually sitting on my desk, lol: it's sparse on the technical details, but I do think the site is still live at https://suffolklitlab.org/resource-map, and the github repo is https://github.com/SuffolkLITLab/resource-map