Python3 Deployment

[ZacMilano]

After the transition to Python3:

• [x] Check scrapy version, make sure that the error with source activate hoaxy-backend && hoaxy --console-log-level=critical crawl --fetch-html --limit=40000 does not reappear
  • [x] This error is caused by a bug of the old version scrapy, when the host name of the crawling HTTPS website is represented as IP address. It is stated in https://github.com/scrapy/scrapy/issues/3029, and fixed in the new version now.
  • [x] check whether the version of scrapy is the newest. And use scrapy shell to test whether we could crawl URL like https://104.238.181.150
• [x] Update master branch, switch production back to master
• [x] Switch production back to burns, restart all cron jobs
• [x] look into disk usage on burns/lenny and for truthy user
• [x] clear up /nobackup, test databases on lenny, back up folders on lenny and burns, etc.
• [x] reboot carl to bring GFS back online
• [x] Update README instructions
  • [x] Document the changes making new system incompatible with Python 2
  • [x] Add requirements for Mercury parser
  • [x] We need to specify/include version for alll the python packages
  • [x] ~1.10 of networkx, else API has issues due to a deprecated method (thanks to Francesco Pierri)~ #30
• [x] Stop test instance on Lenny and clean up the DB
• [x] Once remaining bugs are fixed, parser is updated, docs are updated and AMI is available, contact Francesco (Poly Milano) and Emanuele (ISI/Barcelona) about new version availability (@filmenczer and @glciampaglia)

[glciampaglia]

There may be other people we are not aware of running their own installations; we should probably document in the README the upcoming change, since the new code is likely not going to be compatible with Python 2 anymore.

[filmenczer]

I moved the documentation tasks from issue #20 to here.

[ZacMilano]

We're ignoring this error for now because now it's a warning instead of an error, and we assume it's OK to skip pages with certificate errors.

'Remote certificate is not valid for hostname "104.238.181.150"; Invalid DNS pattern b'104.238.181.150''

[chathuriw]

@filmenczer There are no unnecessary directories in truthy home as I can see. I clean up conda environment which took most of the space.

[truthy@burns ~]$ du -s * | sort -nr 7518352 miniconda3 642524 pylucene 626248 java 496336 code 113524 node_modules 108252 botometer-database 85340 nltk_data 4752 hoaxy-python3 452 2018-07-01.json 192 crontabs 60 hs_err_pid9349.log 60 hs_err_pid8111.log 60 hs_err_pid6872.log 60 hs_err_pid6506.log 60 hs_err_pid572.log 60 hs_err_pid5632.log 60 hs_err_pid5181.log 60 hs_err_pid4001.log 60 hs_err_pid31732.log 60 hs_err_pid30882.log 60 hs_err_pid29809.log 60 hs_err_pid28588.log 60 hs_err_pid27393.log 60 hs_err_pid2723.log 60 hs_err_pid22653.log 60 hs_err_pid21925.log 60 hs_err_pid21658.log 60 hs_err_pid20374.log 60 hs_err_pid18444.log 60 hs_err_pid17394.log 60 hs_err_pid1723.log 60 hs_err_pid16904.log 60 hs_err_pid15690.log 60 hs_err_pid14593.log 52 package-lock.json 44 redis-celery 44 cronjobs 32 auth.iuni.iu.edu 16 pgbouncer 16 botometer-csr 8 2018-07-01-messages.json 4 truthy.keytab 4 test.py 4 README 4 pycache 4 pg_hoaxy.sh 4 ips.txt 0 truthy-website 0 nohup.out 0 hoaxy-frontend 0 hoaxy-backend 0 botometer-website 0 bei

[filmenczer]

@filmenczer will take care of communicating the update as soon as

• [x] @chathuriw resolves the pending Mercury timeout issue.
• [x] The backend will be ready also as an AMI (Amazon Machine Image) for AWS. @marcmccarty will add instructions on the Github README

[marcmccarty]

@filmenczer will communicate/advertise about new (Python3) version and AMI: see instructions at https://github.com/IUNetSci/hoaxy-backend#hoaxy-backend-as-aws-image-ami