Automatically update store.html with latest sims

[mattpen]

We should adapt getSimArray.js so that it regularly updates store.html.

Possible options:

1. migrate it to store.html.
   • This may end up sending too many metadata api requests to the website
2. Change the script so that it automatically writes to store.html instead of stdout and runs on a weekly cron job
   • Sims may not get updated instantly
3. Change the script so that it automatically writes to store.html instead of stdout and runs when build-server does a production deploy.
   • Introduces additional dependencies to a separate repo.

@zepumph - this would be good to look at when time allows, let me know if you have any questions or want to pair.

[zepumph]

From status this morning, people seem to be using office-mix more these days, we should bump the priority on this issue.

[zepumph]

@mattpen I think that either option (2) or (3) would be best. What do you think? Hasn't the project worked pretty hard to keep perennial dependency free? Other developers may not like (3) for this reason.

[mattpen]

OK, yes we should probably keep it out of perennial. I think it would be ideal if it ran on demand instead of on a cron job though. How about this?

Option 3(b): Change the script so that it automatically writes to phet-server:/data/web/static/phetsims/office-mix/store.html. The script would be run on phet-server and would be called by the website by DeployProjectPage.deploy().

[zepumph]

@mattpen and I are going to get together on Friday at 9:15 for this.

[mattpen]

@zepumph and I added a script to automatically synchronize office-mix once a week. The script has been tested and added to the cron job. We are going to check on Monday morning to verify that the cron job worked correctly.

[zepumph]

The cron job is run by phet-admin.

[mattpen]

It does not appear that the cron job ran over the weekend. I'm going to add logging and try again.

[mattpen]

The logs of stdout will now be stored in /home/phet-admin/office-mix-out.log.

[zepumph]

Looks like the cron job had this: /data/share/phet/phet-office-mix/synchronizeOfficeMix.sh (spelled correctly) but the file was called /data/share/phet/phet-office-mix/syncronizeOfficeMix.sh.

I changed the name so all is spelled correctly now. Lets checking next week.

[zepumph]

I see nothing in the log, but perhaps that isn't all bad. Here is the line in ls -la

-rw-r--r--. 1 phet-admin phet 0 Mar 20 01:09 office-mix-out.log

And here is the cron job 9 1 * * 1 /data/share/phet/phet-office-mix/synchronizeOfficeMix.sh >> ~/office-mix-out.log

It looks like it was modified (albeit blank) at the right time.

listing all of /data/web/static/phetims/offixe-mix shows:

phet-admin@phet-server:/data/web/static/phetsims/office-mix$ ls -la
total 380
drwxrwsr-x.  3 tomcat phet   4096 Mar 10 10:09 .
drwxrwsr-x. 18 tomcat phet   8192 Mar 14 18:23 ..
drwxrwsr-x.  2 tomcat phet   4096 Mar 10 09:38 icons
-rw-rw-r--.  1 tomcat phet 185823 Mar 15 17:46 jquery.d.ts
-rw-rw-r--.  1 tomcat phet  16716 Mar 15 17:46 knockout.d.ts
-rw-rw-r--.  1 tomcat phet  73502 Mar 15 17:46 labs-1.0.4.d.ts
-rw-rw-r--.  1 tomcat phet   3431 Mar 15 17:46 site.css
-rw-rw-r--.  1 tomcat phet  45391 Mar 15 17:46 store.html
-rw-rw-r--.  1 tomcat phet   9672 Mar 15 17:46 store.js
-rw-rw-r--.  1 tomcat phet  10814 Mar 15 17:46 store.ts

Those dates modified don't make me feel as good. @mattpen do you have any thoughts as to why there was no log but it didn't work? does >> cover std error as well?

[zepumph]

The permissions of the file look good: -rwxrwxr-x. 1 phet-admin phet 323 Mar 10 10:03 synchronizeOfficeMix.sh

[mattpen]

I think >> only redirects stdout. To redirect stderr you need to do 2>>target. To redirect them both I think you use &>>.

[mattpen]

@zepumph - It looks like it may be a problem with the github credentials. There was no push from the script yesterday recorded.

[mattpen]

I think we may need to setup a git credential store for phet-admin on phet-server in order to actually get the git interaction to work.

@jbphet - If we setup a generic phetadmin account on github, would it be useful to use for the rosetta/babel interaction as well? I think right now it is just using your creds.

[jbphet]

We are using my credentials because when we first set this up the GitHub user account requirements specifically prohibited what it called, "machine accounts". I just looked up the policies, and it looks like this rule may have changed. Take a look at the link below, but what I'm seeing that seems new is the part that says

We do permit machine accounts.

Followed by:

A machine account is an account set up by an individual human who accepts the Terms on behalf of the account, provides a valid email address, and is responsible for its actions. A machine account is used exclusively for performing automated tasks. Multiple users may direct the actions of a machine account, but the owner of the account is ultimately responsible for the machine's actions. You may maintain no more than one free machine account in addition to your free personal account.

I've pasted this information into the issue in case it changes again - we want to have a record of what we based our decision on. Here's the link: https://help.github.com/articles/github-terms-of-service/#2-account-requirements.

So yes, given the updated GitHub policies, I am all for setting up a machine account and using it for rosetta's interactions with babel and any other automated actions that we need, such as the items discussed above.

[zepumph]

Carry on discussion in https://github.com/phetsims/special-ops/issues/66

[zepumph]

There was the same problem Monday night. I changed the cron job to run in 30 minutes, and I got it to append stderr as well.

0 10 * * 2 /data/share/phet/phet-office-mix/synchronizeOfficeMix.sh >> ~/office-mix-out.log 2>>&1

[zepumph]

Alright I did some work and got the logging to work, and the git stuff to work. I used this as my resource to help change permissions. It worked.

http://stackoverflow.com/questions/6448242/git-push-error-insufficient-permission-for-adding-an-object-to-repository-datab

Here is what the crontab looks like now:

# Download any changes to sims and republish them in /office-mix on the phet website.
  30 1 * * 1 /data/share/phet/phet-office-mix/synchronizeOfficeMix.sh >> ~/office-mix-out.log 2>> ~/office-mix-out.log

[zepumph]

Let's checkin next monday and close if all looks good. I also added the date to stdout, so the log will have that.

[mattpen]

I checked today. The office-mix app has unit rates added AND there was a successful git push on Monday. The script ran successfully the previous Monday, but there was no git commit because there were no major/minor/maintenance releases in the week prior.

It appears everything is working as designed. Closing.