Import grants data for the Templeton Foundation into Donations List Website

[vipulnaik]

See https://templeton.org/grants/grant-database for more details

[riceissa]

This is basically done at https://github.com/riceissa/templeton-foundation

As with the other foundations, I left a bunch of FIXMEs in the script where you might want to take a look and change things.

Also there is the "flory" grant duplicates problem. e.g.

• https://templeton.org/grant/religious-competition-and-creative-innovation-los-angeles-and-seoul-south-korea-93
• https://templeton.org/grant/religious-competition-and-creative-innovation-los-angeles-and-seoul-south-korea-92
• https://templeton.org/grant/religious-competition-and-creative-innovation-los-angeles-and-seoul-south-korea-91

[riceissa]

I just sent them an email about this.

[riceissa]

They responded and said the duplicates are not intentional. They said they would get it fixed but didn't say if they would contact me again once fixed.

[riceissa]

They never replied, but it looks like the duplicate grants are gone now (checked "flory" and "thaler" duplicates, and both are gone), and the extraneous grant pages 404.

[riceissa]

Okay I got new data and produced the SQL using both old and new data https://github.com/riceissa/templeton-foundation

• Old data: https://github.com/riceissa/templeton-foundation/blob/master/out-2017-12-30.sql
• New data: https://github.com/riceissa/templeton-foundation/blob/master/out-2018-02-01.sql

It looks like there are changes other than removal of the duplicates, but it's somewhat difficult to view the diff, so I'll have to do it later (perhaps tomorrow).

[riceissa]

I sorted the grants before printing to make the diffing easier, and it turns out that the only changes were those that removed duplicates (so I was wrong about that in the previous comment). I also checked to see if there are any other duplicates, and it seems that every line in the new SQL output is unique.

You can also try:

diff -q <(sort -u out-2017-12-30.sql) <(sort -u out-2018-02-01.sql)

which returns no output, showing that the two files are the same once sorted and uniquified.

[vipulnaik]

Inserted and verified, switching to cleanup_pending and reassigning to myself.