NEAT should check S3 credentials at beginning of run to avoid credential error at the end of a long run

[justaddcoffee]

For example, this happened at the end of a long run (see below).

Should be easy to avoid by checking that ~/.ec2/credentials exists if S3 block is given in NEAT YAML

Epoch 100/100
120/120 [==============================] - 536s 4s/step - loss: 12.9093
Traceback (most recent call last):
  File "/usr/local/bin/neat", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/neat/cli.py", line 68, in run
    upload_dir_to_s3(**upload_kwargs)
  File "/usr/local/lib/python3.7/dist-packages/neat/upload/upload.py", line 21, in upload_dir_to_s3
    client.head_object(Bucket=s3_bucket, Key=s3_path)
  File "/usr/local/lib/python3.7/dist-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/dist-packages/botocore/client.py", line 663, in _make_api_call
    operation_model, request_dict, request_context)
  File "/usr/local/lib/python3.7/dist-packages/botocore/client.py", line 682, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.7/dist-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python3.7/dist-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/usr/local/lib/python3.7/dist-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/usr/local/lib/python3.7/dist-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.7/dist-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.7/dist-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/usr/local/lib/python3.7/dist-packages/botocore/signers.py", line 162, in sign
    auth.add_auth(request)
  File "/usr/local/lib/python3.7/dist-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
Makefile:99: recipe for target 'embedding-upheno-hp-mp-with-relations-with-mp-hp-pistoia' failed
make: *** [embedding-upheno-hp-mp-with-relations-with-mp-hp-pistoia] Error 1
21.302u 5.897s 12:55:57.82 0.0% 0+0k 0+0io 0pf+0w

[justaddcoffee]

Also, we probably should at least warn, probably exit, if the output directory already exists on the remote s3 bucket

[justaddcoffee]

closed by #39