Automate transferring prediction to Oracle

[geneorama]

The predictions will appear alongside the trap map data.

[PriyaDoIT]

• [ ] Gene will place the updating script in a cron job

[tomschenkjr]

@geneorama - please update this thread when the automation is completed.

[nicklucius]

• [ ] @tomschenkjr will create a token for @geneorama to use on the analytics server.
• [ ] @geneorama will then finish the automation

[tomschenkjr]

@geneorama - a slight change if plans. Please generate a new key on the analytics server using "developers@cityofchicago.org" as the email address. Provide it to me, but not on this thread.

[geneorama]

@tomschenkjr generated key in ~/.ssh/, sending in email.

I moved the old keys to ~/.ssh/oldsshkeys (I tested a few options previously)

[tomschenkjr]

@geneorama - key has been added. Please test.

[geneorama]

Had some errors, but now it's working.

In the process of troubleshooting I updating the git global user.name and user.email to developers@cityofchicago.org. I don't know if that was necessary, but I think it's a good idea.

The thing that I think fixed it was updating the location of the ssh key in the .gitconfig file. Previously I was pointing to a file called ~/.ssh/github.

contents of ~/.ssh/config:

Host github.com
    Hostname ssh.github.com
    Port 443
    User git
    IdentityFile ~/.ssh/id_rsa
    IdentitiesOnly yes

[tomschenkjr]

I wouldn’t recommend setting it globally, but perhaps just locally for that repo: https://help.github.com/articles/setting-your-username-in-git/#platform-linux

From: Gene Leynes [mailto:notifications@github.com] Sent: Tuesday, June 06, 2017 11:22 AM To: Chicago/WNV_model WNV_model@noreply.github.com Cc: Schenk, Tom Tom.Schenk@cityofchicago.org; Mention mention@noreply.github.com Subject: Re: [Chicago/WNV_model] Automate transferring prediction to Oracle (#21)

Had some errors, but now it's working.

In the process of troubleshooting I updating the git global user.name and user.email to developers@cityofchicago.orgmailto:developers@cityofchicago.org. I don't know if that was necessary, but I think it's a good idea.

The thing that I think fixed it was updating the location of the ssh key in the .gitconfig file. Previously I was pointing to a file called ~/.ssh/github.

contents of ~/.ssh/config:

Host github.com

Hostname ssh.github.com

Port 443

User git

IdentityFile ~/.ssh/id_rsa

IdentitiesOnly yes

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/Chicago/WNV_model/issues/21#issuecomment-306539825, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABkC0abvJC1rP6GXQ595BUhlQb3k8grhks5sBXyxgaJpZM4LoQxK.

---

This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail (or the person responsible for delivering this document to the intended recipient), you are hereby notified that any dissemination, distribution, printing or copying of this e-mail, and any attachment thereto, is strictly prohibited. If you have received this e-mail in error, please respond to the individual sending the message, and permanently delete the original and any copy of any e-mail and printout thereof.

[geneorama]

Updates:

RGEOS: Balaji installed rgeos... but now ROracle is not found.

GIT / SSH CONFIG: After testing the code in RStudio as me, pushing the changes, and pulling them down again as the service user, I found out that my ssh key wasn't working again for github for developers@cityofchciago.org. Looking at the history I realized that Avishek had reverted the ssh key / git config. I'm guessing that the changes broke his ability to pull the lead repository. So, I changed the ssh config to be able to have different keys for different projects:

Host github.com
    Hostname ssh.github.com
    Port 443
    User git
    IdentityFile ~/.ssh/github
    IdentitiesOnly yes

Host coc-developers-at-github.com
    Hostname ssh.github.com
    Port 443
    User git
    IdentityFile ~/.ssh/id_rsa
    IdentitiesOnly yes

Then edited the project git config to point to the right ssh key:

[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote "origin"]
        url = git@coc-developers-at-github.com:Chicago/WNV_model.git
        fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
        remote = origin
        merge = refs/heads/master
[branch "rversion"]
        remote = origin
        merge = refs/heads/rversion
[user]
        name = SVC-rproject
        email = developers@cityofchicago.org

Also, you can see that I modified the user.name and user.email. I settled on SVC-rproject as the name, previously I was just repeating the email (which might be confusing).

[geneorama]

Installed ROracle for SVC-rproject using R CMD INSTALL --configure-args='--with-oci-lib=/usr/lib/oracle/12.1/client64/lib --with-oci-inc=/usr/include/oracle/12.1/client64' ROracle_1.3-1.tar.gz

(First tried installing within R with install.packages. That failed, but I made a note of the temp file where R put the package, and copied the package to ~)

[geneorama]

The automation is working, but I was doing a little editing to make the output less verbose and clean a few things up.

In the process of cleaning things up I noticed that I am geocoding the ward later when I reshape the data and create the features.

I converted the code to use chigeocodR instead of sp and rgdal, but now I have NA's for the wards for the sites that technically fall outside the city. Apparently the ward map's boundaries are fuzzy enough that they can capture areas that are slightly outside the city.

So, I think it would be better to use the ward map?

[geneorama]

Reverted to using ward map for now, but I also left the code in (in comments) for chigeocodR.

Automation appears to be working, closing issue.

The automation is set to check the data portal every hour. It calculates a hash of the WNV data and compares that to the last hash. If the hash changes, it updates, if not, it exits. This relies on the digest package from Dirk Eddelbuettel.