Data retrieval

[davemacdo]

I don't want to hit the Google Sheets API for each visit. It's slooooow with a giant sheet like this, and it fails a lot.

Future plan:

• Script running on a computer somewhere at regular intervals that would download a copy of the Google Sheet as a JSON file and automatically check it in to the repo, or at least put it in a static location.

If any collaborators want to take a swing at this, I think a simple BASH or PHP script could do the trick.

[gtwright]

Congrats on this project! Very excited to see where it goes.

Have y'all considered using a service like Algolia for search indexing? It's a different approach to speeding things up (and might make it easier to add filters, weight different fields, expand geo search, etc.) They've got a solid free plan and will sponsor open source or non-profit projects with a higher level of use. Plus Zapier has a (beta) Algolia integration that can pull new/updated Google Sheet rows into the search index.

[davemacdo]

@gtwright That's an interesting idea. Currently, the app loads the whole table in a big JSON file and searches/filters locally. I don't have any experience working with an external service like Algolia. I have been contemplating moving everything to Google Firestore, making a friendlier backend for the research team, and getting rid of the spreadsheet entirely. In Firestore, I would consolidate many of the boolean fields into grouped arrays, which I'm hoping will speed things up.

I'm certain you know more about this stuff than me. Do you think a search service would be a better solution? Either way, I'm teaching myself everything as I go (which is probably painfully obvious if you've looked at any of the source!).

[gtwright]

I'll never judge anyone else's code—I'm pretty self-taught myself :) I'd say you've done some great work here.

Making a move away from Google Sheets seems like an excellent investment of time. I'm sure Firestore would work well. You might also do well in looking for a CMS that would be fairly user-friendly for the research team. Assuming you're sticking more or less with the kind of interface you have now (searching/faceting/sorting), you've got a lot of options for the database side. Anything that spits out JSON should do. (Even headless Wordpress might do the trick)

As far as Algolia (or its competitors) goes: it's sorta made just for this kind of thing. And it's not too hard to pick up. The speed is ridiculous, but it also lets you have things like typo tolerance, weighted search priorities, geo-fencing, etc. Here's an example: https://community.algolia.com/react-instantsearch/examples/e-commerce-infinite/ The tutorials are solid.

I'm happy to chat more about where I could help out. I'm guessing this list of issues/enhancements doesn't quite sum up all you've got planned.

[davemacdo]

Thanks! I think Firestore will be a good first step, and it may make it easier to incorporate a search service like Algolia down the road to have the data in a more flexible structure. Either way, the JSON will need some similar converting. Currently, each of the select boxes represents a field in the database that is either 'X' or an empty string. I think a better structure would be to treat them more like tags, and have an array of strings for each category (genre, subgenre, etc.) so that each record will have six or seven fields instead of forty-plus.

There is only a little bit more to the plan (for software, anyhow) than is here. As soon as I figure out the details, I'll be turning them into issues here. I will close this issue as a wontfix since the scope is changing and post some new issue enhancement threads in the next few days. I'll start an issue for discussing the data structure first, and hopefully the best way forward can emerge from that.

[gtwright]

I'm with you 100% on all of this—especially focusing on a data structure conversation before moving anything else forward.

By the way, if you ever want to get ahold of me outside of a discussion thread here, my email is graham@opusaffair.com.

[captbaritone]

Hey! Cool project. Thanks for making it open source. Regarding CMS, https://github.com/netlify/netlify-cms might be a good fit, but I’ve not used it personally.

[davemacdo]

Thanks for the suggestions, @gtwright and @captbaritone. Since the scope of this issue has changed, I'm closing it and replacing with these:

• New data structure, preparing database move
• Moving data

I'm very new to Vue, open source, and pretty much everything related to this project, so I'm very open to feedback. If interested, you can contact me outside GH Issues at davidjohnmacdonald@gmail.com.