Data Download - Data automatically recompiled every 2 weeks for download

[alexchippendale]

We are in the process of building the generation system that will automatically recompile the data every 2 weeks for download.

[SecondStreetDevelopment]

Yes, I'm working on this right now. @jkhartshorne, feel free to contact me if you have any specific requests for this feature.

[jkhartshorne]

The dmgs that I've been getting have been perfect.

I understand why it might make sense to do only info that is new or changed since the last download, but there are many ways for that to go wrong. So I prefer the simplicity of just dumping everything.

[SecondStreetDevelopment]

@jkhartshorne, thank you, that is good to know. I will talk with David and get a copy of what he has been sending you and duplicate that.

[alexchippendale]

@jkhartshorne The code for this is complete now, we will be activating it after we update production.

[jkhartshorne]

Awesome!

[SecondStreetDevelopment]

Hey @jkhartshorne, I just manually executed the cloud function which generates the bi-weekly download. You should receive an email with a link. Let me know if you see it and if there are any problems.

[jkhartshorne]

I emailed @lorenachipstudio a few hours ago, but in case she didn't pass this on:

This is great EXCEPT it doesn't include the most important json -- the one with the list of actual recordings! it's ALSO missing the list of guesses for "Say What?". Please add that as well.

[jkhartshorne]

@lorenachipstudio @alexchippendale @SecondStreetDevelopment - I wanted to make sure you saw this. I really could use those data. I haven't had an update since the first week of August. (As I told Lorena, the one she sent a week or so ago didn't work.)

[SecondStreetDevelopment]

@jkhartshorne, I'm working on the corrections to this feature right now. I'm not sure why those two json's were omitted, they were generated without issue on our dev site but some difference in the production site (perhaps the amount of data) broke things. I should have something for you very soon.

[SecondStreetDevelopment]

It looks like the amount of data included in the download has caused us to run into a bug within Firebase. The Firebase devs have not produced a fix for this bug and it's been over a year. I'm working on a multi-cloud function way to generate this data download so we can side-step this bug.

[jkhartshorne]

Don't you hate it when that happens? We had a similar bug with Linux for Microsoft on a different project.

Any chance you can send me a download directly in the meantime? We're really anxious to see how some recent work we've been doing has impacted participation (or not). It's been almost 5 weeks since our last data dump.

Thanks! Josh

On Fri, Sep 11, 2020 at 11:11 AM Mitchell notifications@github.com wrote:

It looks like the amount of data included in the download has caused us to run into a bug within Firebase. The Firebase devs have not produced a fix for this bug and it's been over a year. I'm working on a multi-cloud function way to generate this data download so we can side-step this bug.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/alexchippendale/KidTalkRemainingTasks/issues/16#issuecomment-691152953, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOVVY2EPUQU6ZMSCFEOLC3SFI4Y7ANCNFSM4PZQQZXQ .

[SecondStreetDevelopment]

@jkhartshorne, I just sent an export from yesterday to your email. I will followup with the missing guesses json soon.

[jkhartshorne]

Thanks!

[jkhartshorne]

I don't think I got the latest email? Can you tell me what email address it comes from and I can double-check?

[SecondStreetDevelopment]

It should come from no-reply@kidtalk.app. It's setup to automatically send twice a month. I'm still finishing up some tweaks so the latest email may be missing some jsons. If so, just disregard and let me know, I will manually send you the complete export.

[jkhartshorne]

I checked. I haven't gotten anything from no-reply@kidtalk.app in the last three weeks.

[SecondStreetDevelopment]

I'll check the logs and let you know. Yesterday, we discussed modifications to make this function run better. Once I have these mods completed I'll manually initiate the function and then follow up to make sure you received it.

[SecondStreetDevelopment]

@jkhartshorne, these modifications have been published to production. I broke the function into two recurring functions, one for the database export, and one for the recordings. This was necessary in order to make sure we did not run into the execution time limitations of the platform. Both functions will run automatically on the 1st and the 16th of each month. The execution on the first will contain a download of the recordings from the previous month as well as the entire database. The execution on the 16th will include recordings from the 1st through the 15th of the current month as well as the entire database. It was necessary to limit the recordings to just the current or previous month to reduce function execution time. I have just manually initiated the recurring functions. You should have received three emails, one with the September recordings, one with the current October recordings, and one with the entire database. Let me know if you have not received the emails or if there are any issues.

[SecondStreetDevelopment]

I've added a drop-down to the admin page so you can view the previous data exports (recordings/collections) and download them: