openaustralia / yinyo

A wonderfully simple API driven service to reliably execute many long running scrapers in a super scaleable way
https://yinyo.io
Apache License 2.0
6 stars 1 forks source link

client.sh hangs getting event data from clay #65

Closed jamezpolley closed 4 years ago

jamezpolley commented 4 years ago

The run_token provided by Clay isn't recognised when later sent back to Clay.

Output from client.sh (modified to add -x to shebang line)

++ create test/scrapers/test-python
++ curl -s -G -X POST http://localhost:8080/runs -d name_prefix=test/scrapers/test-python
+ create_result='{"run_name":"test-scrapers-test-python-fqsjx","run_token":"VfoS9cvH8ZJ3VFyUBSpefimlLwejpS06"}'
++ echo '{"run_name":"test-scrapers-test-python-fqsjx","run_token":"VfoS9cvH8ZJ3VFyUBSpefimlLwejpS06"}'
++ jq -r .run_name
+ run_name=test-scrapers-test-python-fqsjx
++ echo '{"run_name":"test-scrapers-test-python-fqsjx","run_token":"VfoS9cvH8ZJ3VFyUBSpefimlLwejpS06"}'
++ jq -r .run_token
+ run_token=VfoS9cvH8ZJ3VFyUBSpefimlLwejpS06
++ pwd
+ dir=/home/james/src/oaf/morph-ng/morph-ng
+ cd test/scrapers/test-python
+ put test-scrapers-test-python-fqsjx VfoS9cvH8ZJ3VFyUBSpefimlLwejpS06 app
+ curl -s -X PUT -H 'Authorization: Bearer VfoS9cvH8ZJ3VFyUBSpefimlLwejpS06' --data-binary @- --no-buffer http://localhost:8080/runs/test-scrapers-test-python-fqsjx/app
+ tar -zcf - data.sqlite requirements.txt runtime.txt scraper.py
+ cd /home/james/src/oaf/morph-ng/morph-ng
+ cache_path=assets/client-storage/cache/test/scrapers/test-python.tgz
+ '[' -f assets/client-storage/cache/test/scrapers/test-python.tgz ']'
+ start test-scrapers-test-python-fqsjx VfoS9cvH8ZJ3VFyUBSpefimlLwejpS06 data.sqlite SCRAPER_NAME test/scrapers/test-python ''
+ '[' '' == '' ']'
++ jq -c -n --arg output data.sqlite --arg env_name SCRAPER_NAME --arg env_value test/scrapers/test-python '{output: $output, env: [{name: $env_name, value: $env_value}]}'
+ data='{"output":"data.sqlite","env":[{"name":"SCRAPER_NAME","value":"test/scrapers/test-python"}]}'
+ curl -s -X POST -H 'Authorization: Bearer VfoS9cvH8ZJ3VFyUBSpefimlLwejpS06' -H 'Content-Type: application/json' http://localhost:8080/runs/test-scrapers-test-python-fqsjx/start -d '{"output":"data.sqlite","env":[{"name":"SCRAPER_NAME","value":"test/scrapers/test-python"}]}'
+ '[' VfoS9cvH8ZJ3VFyUBSpefimlLwejpS06 = '' ']'
+ events test-scrapers-test-python-fqsjx VfoS9cvH8ZJ3VFyUBSpefimlLwejpS06
+ curl -s --no-buffer -H 'Authorization: Bearer VfoS9cvH8ZJ3VFyUBSpefimlLwejpS06' http://localhost:8080/runs/test-scrapers-test-python-fqsjx/events

Output from clay-server:

[clay-server-659cf8c677-744pm clay-server] main.go:201: POST /runs/test-scrapers-test-python-fqsjx/start
[clay-server-659cf8c677-744pm clay-server] main.go:201: GET /runs/test-scrapers-test-python-fqsjx/events
[clay-server-659cf8c677-744pm clay-server] main.go:201: POST /runs/test-scrapers-test-python-fqsjx/events
[clay-server-659cf8c677-744pm clay-server] main.go:201: GET /runs/test-scrapers-test-python-fqsjx/app
[clay-server-659cf8c677-744pm clay-server] main.go:252: The access key ID you provided does not exist in our records.

Job output:

runName test-scrapers-test-python-fqsjx
runToken VfoS9cvH8ZJ3VFyUBSpefimlLwejpS06
runOutput data.sqlite
serverURL http://clay-server.clay-system:8080
buildCommand /bin/herokuish buildpack build
runCommand /bin/herokuish procfile start scraper
root.go:171: 500 Internal Server Error
jamezpolley commented 4 years ago

\

jamezpolley commented 4 years ago

I can't figure out what's happening, or what's meant to be happening, so I can't file a useful bug.