KeyError: 'access_token' upon uploading large amounts of subjects

[khufkens]

Upon uploading a large amount of data to a subject set the upload would halt on this error. Subsequently an yaml error file was generated but I had to restart the upload to continue.

This consistently happens around the 40K values mark, despite not exceeding subject upload limits.

Traceback (most recent call last):
  File "/home/khufkens/venv/bin/panoptes", line 8, in <module>
    sys.exit(cli())
  File "/home/khufkens/venv/lib/python3.5/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/khufkens/venv/lib/python3.5/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/khufkens/venv/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/khufkens/venv/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/khufkens/venv/lib/python3.5/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/khufkens/venv/lib/python3.5/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/khufkens/venv/lib/python3.5/site-packages/panoptes_cli/commands/subject_set.py", line 401, in upload_subjects
    move_created(MAX_PENDING_SUBJECTS)
  File "/home/khufkens/venv/lib/python3.5/site-packages/panoptes_cli/commands/subject_set.py", line 372, in move_created
    if subject.async_save_result:
  File "/home/khufkens/venv/lib/python3.5/site-packages/panoptes_client/subject.py", line 203, in async_save_result
    self._async_future.result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/khufkens/venv/lib/python3.5/site-packages/panoptes_client/subject.py", line 143, in save
    log_args=False,
  File "/home/khufkens/venv/lib/python3.5/site-packages/redo/__init__.py", line 185, in retry
    return action(*args, **kwargs)
  File "/home/khufkens/venv/lib/python3.5/site-packages/panoptes_client/panoptes.py", line 815, in save
    etag=self.etag
  File "/home/khufkens/venv/lib/python3.5/site-packages/panoptes_client/panoptes.py", line 404, in post
    retry=retry,
  File "/home/khufkens/venv/lib/python3.5/site-packages/panoptes_client/panoptes.py", line 267, in json_request
    retry=retry,
  File "/home/khufkens/venv/lib/python3.5/site-packages/panoptes_client/panoptes.py", line 191, in http_request
    token = self.get_bearer_token()
  File "/home/khufkens/venv/lib/python3.5/site-packages/panoptes_client/panoptes.py", line 557, in get_bearer_token
    self.bearer_token = token_response['access_token']
KeyError: 'access_token'

[TECOLOGYxyz]

I am getting a similar error. I am also uploading a large number of images, but I am not close to the limit. The upload fails and when I generate the yaml, I get the error below. A yaml is generated, but I am not sure, I can trust that it is completed?

File "/home/bitcue/anaconda3/envs/zooniverse/lib/python3.9/site-packages/panoptes_cli/commands/subject_set.py", line 412, in upload_subjects
    link_subjects(LINK_BATCH_SIZE)
  File "/home/bitcue/anaconda3/envs/zooniverse/lib/python3.9/site-packages/panoptes_cli/commands/subject_set.py", line 390, in link_subjects
    subject_set.add(list(upload_state['waiting_to_link'].keys()))
  File "/home/bitcue/anaconda3/envs/zooniverse/lib/python3.9/site-packages/panoptes_client/subject_set.py", line 95, in add
    return self.links.subjects.add(subjects)
  File "/home/bitcue/anaconda3/envs/zooniverse/lib/python3.9/site-packages/panoptes_client/subject_set.py", line 40, in add
    return super(SubjectSetLinkCollection, self).add(objs)
  File "/home/bitcue/anaconda3/envs/zooniverse/lib/python3.9/site-packages/panoptes_client/utils.py", line 56, in do_batch
    func(_self, batch, *args, **kwargs)
  File "/home/bitcue/anaconda3/envs/zooniverse/lib/python3.9/site-packages/panoptes_client/panoptes.py", line 1037, in add
    _objs = [obj for obj in self._build_obj_list(objs) if obj not in self]
  File "/home/bitcue/anaconda3/envs/zooniverse/lib/python3.9/site-packages/panoptes_client/panoptes.py", line 1037, in <listcomp>
    _objs = [obj for obj in self._build_obj_list(objs) if obj not in self]
  File "/home/bitcue/anaconda3/envs/zooniverse/lib/python3.9/site-packages/panoptes_client/subject_set.py", line 25, in __contains__
    linked_subject_count = SetMemberSubject.where(
  File "/home/bitcue/anaconda3/envs/zooniverse/lib/python3.9/site-packages/panoptes_client/panoptes.py", line 672, in where
    return cls.paginated_results(*cls.http_get(_id, params=kwargs))
  File "/home/bitcue/anaconda3/envs/zooniverse/lib/python3.9/site-packages/panoptes_client/panoptes.py", line 618, in http_get
    return Panoptes.client().get(
  File "/home/bitcue/anaconda3/envs/zooniverse/lib/python3.9/site-packages/panoptes_client/panoptes.py", line 314, in get
    return self.json_request(
  File "/home/bitcue/anaconda3/envs/zooniverse/lib/python3.9/site-packages/panoptes_client/panoptes.py", line 259, in json_request
    response = self.http_request(
  File "/home/bitcue/anaconda3/envs/zooniverse/lib/python3.9/site-packages/panoptes_client/panoptes.py", line 191, in http_request
    token = self.get_bearer_token()
  File "/home/bitcue/anaconda3/envs/zooniverse/lib/python3.9/site-packages/panoptes_client/panoptes.py", line 557, in get_bearer_token
    self.bearer_token = token_response['access_token']

[adammcmaster]

You can trust the YAML file. It contains the internal state at the point the error was encountered, so the upload can be resumed in such a way as to guarantee everything is uploaded correctly.