HTTPError: 504 Server Error: Gateway Time-out for url

[sgbaird]

Hmm, I'm getting this after ~7 minutes of running the cell.

HTTPError                                 Traceback (most recent call last)
Cell In[3], line 29
     27 key = f"matsciopt-{task_name}-benchmark-dataset"
     28 access_token = ZENODO_SANDBOX_API_KEY if sandbox else ZENODO_API_KEY
---> 29 res = ensure_zenodo(
     30     key,
     31     data=data,
     32     paths=[
     33         f"../../data/processed/{task_name_underscore}/sobol_probability_filter.csv",
     34         f"../../data/processed/{task_name_underscore}/sobol_regression.csv",
     35         f"../../data/processed/{task_name_underscore}/model_metadata.json",
     36         f"../../models/{task_name_underscore}/surrogate_models.pkl",
     37         f"../../models/cv/{task_name_underscore}/cross_validation_models_0.pkl",
     38         f"../../models/cv/{task_name_underscore}/cross_validation_models_1.pkl",
     39         f"../../models/cv/{task_name_underscore}/cross_validation_models_2.pkl",
     40         f"../../models/cv/{task_name_underscore}/cross_validation_models_3.pkl",
     41         f"../../models/cv/{task_name_underscore}/cross_validation_models_4.pkl",
     42     ],
     43     sandbox=sandbox,  # remove this when you're ready to upload to real Zenodo
     44     access_token=access_token,
     45 )
     46 from pprint import pprint
     48 pprint(res.json())

File c:\Users\sterg\Miniconda3\envs\matsci-opt-benchmarks\lib\site-packages\zenodo_client\api.py:37, in ensure_zenodo(key, data, paths, **kwargs)
     35 def ensure_zenodo(key: str, data: Data, paths: Paths, **kwargs) -> requests.Response:
     36     """Create a Zenodo record if it doesn't exist, or update one that does."""
---> 37     return Zenodo(**kwargs).ensure(key=key, data=data, paths=paths)

File c:\Users\sterg\Miniconda3\envs\matsci-opt-benchmarks\lib\site-packages\zenodo_client\api.py:99, in Zenodo.ensure(self, key, data, paths)
     97 if deposition_id is not None:
     98     logger.info("mapped local key %s to deposition %s", key, deposition_id)
---> 99     return self.update(deposition_id=deposition_id, paths=paths)
    101 res = self.create(data=data, paths=paths)
    102 # Write the ID to the key in the local configuration
    103 # so it doesn't need to be created from scratch next time

File c:\Users\sterg\Miniconda3\envs\matsci-opt-benchmarks\lib\site-packages\zenodo_client\api.py:195, in Zenodo.update(self, deposition_id, paths)
    191 bucket = new_deposition_data["links"]["bucket"]
    193 # Upload new files. It calculates the hash on all of these, and if no files have changed,
    194 #  there will be no update
--> 195 self._upload_files(bucket=bucket, paths=paths)
    197 # Send the publish command
    198 return self.publish(new_deposition_id)

File c:\Users\sterg\Miniconda3\envs\matsci-opt-benchmarks\lib\site-packages\zenodo_client\api.py:212, in Zenodo._upload_files(self, bucket, paths)
    205     with open(path, "rb") as file:
    206         res = requests.put(
    207             f"{bucket}/{os.path.basename(path)}",
    208             data=file,
    209             params={"access_token": self.access_token},
    210         )
--> 212     res.raise_for_status()
    213     rv.append(res)
    214 return rv

File c:\Users\sterg\Miniconda3\envs\matsci-opt-benchmarks\lib\site-packages\requests\models.py:1021, in Response.raise_for_status(self)
   1016     http_error_msg = (
   1017         f"{self.status_code} Server Error: {reason} for url: {self.url}"
   1018     )
   1020 if http_error_msg:
-> 1021     raise HTTPError(http_error_msg, response=self)

HTTPError: 504 Server Error: Gateway Time-out for url: https://zenodo.org/api/files/38a5a2f8-762b-4393-89d1-68edbdbdc90d/model_metadata.json?access_token=<access_token>

Some precedence on the zenodo page https://github.com/zenodo/zenodo/issues/2131 https://github.com/zenodo/zenodo/issues/934

[cthoyt]

Can you give some extra context on how this is relevant for this package versus a more general Zenodo issue?

[sgbaird]

Sure, I think there's a way to increase the timeout, but I don't think zenodo-client has that option. My fallback, for now, will be manually uploading the files for that repo. Maybe there's another option that could skip the checksum, or compute that locally before sending some kind of override (not sure if Zenodo API allows for this).