client.sh does not return.

jamezpolley commented 4 years ago

Expected behabviour: Something happens

Observed behaviour:

$ ./client.sh test/scrapers/test-python data.sqlite

client.sh does not return.

Logs from clay-server:

[clay-server-659cf8c677-744pm clay-server] main.go:201: POST /runs?name_prefix=test/scrapers/test-python
[clay-server-659cf8c677-744pm clay-server] main.go:201: PUT /runs/test-scrapers-test-python-f2ztm/app
[clay-server-659cf8c677-744pm clay-server] main.go:252: The access key ID you provided does not exist in our records.
[clay-server-659cf8c677-744pm clay-server] main.go:201: POST /runs/test-scrapers-test-python-f2ztm/start
[clay-server-659cf8c677-744pm clay-server] main.go:201: GET /runs/test-scrapers-test-python-f2ztm/events
[clay-server-659cf8c677-744pm clay-server] main.go:201: POST /runs/test-scrapers-test-python-f2ztm/events
[clay-server-659cf8c677-744pm clay-server] main.go:201: GET /runs/test-scrapers-test-python-f2ztm/app
[clay-server-659cf8c677-744pm clay-server] main.go:252: The access key ID you provided does not exist in our records.

It seems that the job is failing. Logs from the pod:

container clay-server is not valid for pod test-scrapers-test-python-f2ztm-pqpbh

jamezpolley commented 4 years ago

I think that what's happening here is that the scraper is failing; but clay-server seems to be continuing to wait for more output rather than closing the connection to the client. Because the server doesn't close the connection, the script can't move on.

If that's right, this issue should probably be about clay-server sending an EOF if the job has failed or something like that.

mlandauer commented 4 years ago

"The access key ID you provided does not exist in our records." looks to me like the problem is due to the minio bucket not being created. This can easily be fixed by running make buckets.

However, I do think that there is a lot that could be improved here about the user experience.

67 would stop the clay server from starting up if there is no bucket configured which would be a much clearer failure who's origin is the configuration of the clay server.
68 would improve the error handling of the client, which is currently non-existent. Currently if any of the API calls fail it just carries on regardless which is "not great".

jamezpolley commented 4 years ago

This can easily be fixed by running make buckets.

Yep, that fixed it.

I guess I missed that step the last time I recreated minikube

mlandauer commented 4 years ago

That's great @jamezpolley. In that case I'll close this as fixed in favour of the improvements in #67 and #68.

openaustralia / yinyo

client.sh does not return. #66

67 would stop the clay server from starting up if there is no bucket configured which would be a much clearer failure who's origin is the configuration of the clay server.

68 would improve the error handling of the client, which is currently non-existent. Currently if any of the API calls fail it just carries on regardless which is "not great".