Missing data dump

[michaelaye]

Hi! can it be that with all the database stress for the new site somebody accidentally deleted the regular CSV dump for PlanetFour? Because I didn't get an email about it yesterday as I usually do on Sundays.

[mschwamb]

This might be a question for @astopy and @parrish

[parrish]

We've had some issues with our data export machine running out memory trying to compress files. @astopy It looks like this is still failing.

[adammcmaster]

@parrish Looks like you've run this successfully... is that permanently fixed or do we still need to do something to get it working long-term?

[parrish]

Permanently-ish. The data processing instance is still using our old deployment setup. Also, adding gem dependencies (pg) that required native libraries to be installed required the instance to be rebuilt.

[michaelaye]

Hi, could we have another dump please? Cheers.

[adammcmaster]

This should be fixed now.

[michaelaye]

Just to tune my expectation, will the dump arrive on Tuesdays now like yesterday?

[adammcmaster]

It should usually still arrive on Sunday/early Monday, but the next one might be a little later if it doesn't run on schedule like it's supposed to. (I'll be checking it to make sure it runs, and I'll start it manually if not.)

[michaelaye]

Last dump has not been sent out again?

[adammcmaster]

Oops, I made a typo in the schedule and accidentally set it to run every five years instead of every week (as it turns out, an easier mistake than you’d think). It should be working now.

On 5 Aug 2015, at 01:53, Chris Snyder notifications@github.com wrote:

Assigned #144 https://github.com/zooniverse/MarsZoo/issues/144 to @astopy https://github.com/astopy.

— Reply to this email directly or view it on GitHub https://github.com/zooniverse/MarsZoo/issues/144#event-373666303.

[michaelaye]

You got me intrigued! I'm doing a lot of time based calculations and can not come up with a time coordinate system where a typo could be making 5 years equivalent to 1 week=7 days ... Unless you have to do things in hours and you added a digit after 7*24=168 ? 1680 would be 4.6 years, so, close to the 5 years you stated?

Sent from my iPhone

On Aug 5, 2015, at 05:49, Adam McMaster notifications@github.com wrote:

Oops, I made a typo in the schedule and accidentally set it to run every five years instead of every week (as it turns out, an easier mistake than you’d think). It should be working now.

On 5 Aug 2015, at 01:53, Chris Snyder notifications@github.com wrote:

Assigned #144 https://github.com/zooniverse/MarsZoo/issues/144 to @astopy https://github.com/astopy.

— Reply to this email directly or view it on GitHub https://github.com/zooniverse/MarsZoo/issues/144#event-373666303.

— Reply to this email directly or view it on GitHub.

[adammcmaster]

Hah, well the scheduling system has a way to randomise the precise times that tasks run to avoid everything running at once. The schedule is written similar to the Unix cron system, if you’re familiar with that, where there are columns for minutes, hours, day, month, and day of week. A “H” in a column means that it generates a random value (or “hash”), and a “” in a column means it ignores that column. I wanted to set it to run every Sunday, so the schedule should have been “H H * \ 0” for a random minute/hour on every Sunday, but I accidentally typed “H H H H 0” which means it generated a specific day and month too (in this case it generated November 6 if I remember correctly), so it would only run on November 6 and only when it happened to fall on a Sunday.

With that config it would have last run in 2011 and next in 2016, though now that I check the next year after that would have been 2022 so it would have been even longer than five years.

On 5 Aug 2015, at 16:13, K.-Michael Aye notifications@github.com wrote:

You got me intrigued! I'm doing a lot of time based calculations and can not come up with a time coordinate system where a typo could be making 5 years equivalent to 1 week=7 days ... Unless you have to do things in hours and you added a digit after 7*24=168 ? 1680 would be 4.6 years, so, close to the 5 years you stated?

Sent from my iPhone

On Aug 5, 2015, at 05:49, Adam McMaster notifications@github.com wrote:

Oops, I made a typo in the schedule and accidentally set it to run every five years instead of every week (as it turns out, an easier mistake than you’d think). It should be working now.

On 5 Aug 2015, at 01:53, Chris Snyder notifications@github.com wrote:

Assigned #144 https://github.com/zooniverse/MarsZoo/issues/144 to @astopy https://github.com/astopy.

— Reply to this email directly or view it on GitHub https://github.com/zooniverse/MarsZoo/issues/144#event-373666303.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub https://github.com/zooniverse/MarsZoo/issues/144#issuecomment-128032818.

[michaelaye]

Ah the cron jobs, yes they are special. ;) thanks for sharing!

Sent from my iPhone

On Aug 6, 2015, at 03:41, Adam McMaster notifications@github.com wrote:

Hah, well the scheduling system has a way to randomise the precise times that tasks run to avoid everything running at once. The schedule is written similar to the Unix cron system, if you’re familiar with that, where there are columns for minutes, hours, day, month, and day of week. A “H” in a column means that it generates a random value (or “hash”), and a “” in a column means it ignores that column. I wanted to set it to run every Sunday, so the schedule should have been “H H * \ 0” for a random minute/hour on every Sunday, but I accidentally typed “H H H H 0” which means it generated a specific day and month too (in this case it generated November 6 if I remember correctly), so it would only run on November 6 and only when it happened to fall on a Sunday.

With that config it would have last run in 2011 and next in 2016, though now that I check the next year after that would have been 2022 so it would have been even longer than five years.

On 5 Aug 2015, at 16:13, K.-Michael Aye notifications@github.com wrote:

You got me intrigued! I'm doing a lot of time based calculations and can not come up with a time coordinate system where a typo could be making 5 years equivalent to 1 week=7 days ... Unless you have to do things in hours and you added a digit after 7*24=168 ? 1680 would be 4.6 years, so, close to the 5 years you stated?

Sent from my iPhone

On Aug 5, 2015, at 05:49, Adam McMaster notifications@github.com wrote:

Oops, I made a typo in the schedule and accidentally set it to run every five years instead of every week (as it turns out, an easier mistake than you’d think). It should be working now.

On 5 Aug 2015, at 01:53, Chris Snyder notifications@github.com wrote:

Assigned #144 https://github.com/zooniverse/MarsZoo/issues/144 to @astopy https://github.com/astopy.

— Reply to this email directly or view it on GitHub https://github.com/zooniverse/MarsZoo/issues/144#event-373666303.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub https://github.com/zooniverse/MarsZoo/issues/144#issuecomment-128032818.

— Reply to this email directly or view it on GitHub.