Closed fteufel closed 1 month ago
Thanks for reporting, @fteufel !
I think this would be fixed by #146. It's because we use the dataset name, rather than the slug (i.e. only lowercase letters and dashes).
Uploading Zarr is a multi-stage process, so it can happen that you do see the dataset on the Hub already, even though a later step in the upload process fails. However, you should always see a banner at the top stating that the dataset upload has not completed yet.
Could you try upgrading Polaris to the latest version and let me know if the issue persists?
It worked now for another dataset.
The one that failed at upload was broken, with the banner being there for 16+ hours. I deleted it in the UI in order to upload it again. Now I get
"message": "Dataset 'bend-chromatin-accessibility', with slug 'bend-chromatin-accessibility', already exists"
Do I need to refresh something to make the delete be registered?
@fteufel The reason is the metadata is already registered in the database. I just removed it from the database. You can try again.
Still something hanging
2024-07-22 16:36:47.580 | INFO | polaris._mixins:md5sum:27 - Computing the checksum. This can be slow for large datasets.
Finding all files in the Zarr archive: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1027/1027 [00:00<00:00, 2661.58it/s]
💥 ERROR: Failed to upload dataset.
Traceback (most recent call last):
File "/novo/users/fegt/BEND/scripts/upload_polaris_datasets.py", line 243, in <module>
dataset.upload_to_hub(owner='mlls')
File "/novo/users/fegt/miniconda3/envs/bend/lib/python3.10/site-packages/polaris/dataset/_dataset.py", line 377, in upload_to_hub
self.client.upload_dataset(self, access=access, owner=owner)
File "/novo/users/fegt/miniconda3/envs/bend/lib/python3.10/site-packages/polaris/hub/client.py", line 581, in upload_dataset
hub_response.raise_for_status()
File "/novo/users/fegt/miniconda3/envs/bend/lib/python3.10/site-packages/httpx/_models.py", line 761, in raise_for_status
raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '405 Method Not Allowed' for url 'https://polarishub.io/storage/dataset/mlls/bend-chromatin-accessibility/table.parquet'
@fteufel We are working on a solution to enable dataset updates and aim to make it available as soon as possible. For now, another quick solution for this is to change your dataset name.
Hey there! I'm going to close this issue because it's no longer related to the original issue that was raised.
But before I do, I wanted to provide some context and hopefully solve your issue in the process @fteufel: We made the decision to implement deletion on the Hub as a soft-delete: You won't see deleted content on the Hub anymore, but the actual files still exist. That way we could restore deleted artifacts if needed.
This does imply, however, that you cannot create a dataset with the same name as a previously deleted artifact. In an effort to unblock you quickly, @zhu0619 manually hard-deleted the artifact from our database (solving this), but did not yet delete the associated files that had already completed uploading (causing this).
So where do we go next?
This situation is an exceptional case. Moving forward, we never want to manually delete content because such a manual process is too error-prone and risks our data integrity. For just this once, however, given that we already deleted the entry from our database, we decided to also manually delete the associated, orphaned files from our storage backend. @fteufel This implies that uploading your dataset should work now!
If it doesn't, however, please reach out over Discord! Given that it's such an exceptional case, that's the better place to get personal support.
Ok, understand now - would be great to have that spelled out in the confirmation popup you get when deleting.
But it failed again :(
⠹ Uploading dataset...2024-07-23 09:55:01.849 | INFO | polaris.hub.client:upload_dataset:602 - Copying Zarr archive to the Hub. This may take a while.
💥 ERROR: Failed to upload dataset.
Traceback (most recent call last):
File "/novo/users/fegt/BEND/scripts/upload_polaris_datasets.py", line 243, in <module>
dataset.upload_to_hub(owner='mlls')
File "/novo/users/fegt/miniconda3/envs/bend/lib/python3.10/site-packages/polaris/dataset/_dataset.py", line 377, in upload_to_hub
self.client.upload_dataset(self, access=access, owner=owner)
File "/novo/users/fegt/miniconda3/envs/bend/lib/python3.10/site-packages/polaris/hub/client.py", line 603, in upload_dataset
zarr.copy_store(
File "/novo/users/fegt/miniconda3/envs/bend/lib/python3.10/site-packages/zarr/convenience.py", line 756, in copy_store
dest[dest_key] = data
File "/novo/users/fegt/miniconda3/envs/bend/lib/python3.10/site-packages/zarr/storage.py", line 1470, in __setitem__
self.map[key] = value
File "/novo/users/fegt/miniconda3/envs/bend/lib/python3.10/site-packages/fsspec/mapping.py", line 175, in __setitem__
self.fs.pipe_file(key, maybe_convert(value))
File "/novo/users/fegt/miniconda3/envs/bend/lib/python3.10/site-packages/polaris/hub/polarisfs.py", line 223, in pipe_file
raise PolarisHubError("Could not get signed URL from Polaris Hub.")
polaris.utils.errors.PolarisHubError: Could not get signed URL from Polaris Hub.
@fteufel I see some data made it to our storage backend. I'm not sure why the upload failed midway through, but my best explanation is that your login expired as the upload happened or that you hit some timeout. You're helping us stress test the system here - The downside of working mostly with small molecules is that most datasets I'm used to are small! 😅
Lacking a formal retry mechanism, could you complete the upload using:
First, refresh your login token:
polaris login --overwrite
Then:
from polaris.hub.client import PolarisHubClient
# Create the exact same dataset locally again
dataset = ...
with PolarisHubClient() as client:
# Increase the timeout
client.settings.default_timeout = (100, 2000)
# Open the destination Zarr archive again
dest = client.open_zarr_file(
owner="mlls",
name="bend-chromatin-accessibility",
path="polarisfs://data.zarr",
mode="w",
as_consolidated=False,
)
# Copy the files to the destination, skipping any files that already exist.
# With this code, you will also see additional print output during the process that may help us debug if it fails again.
logger.info("Copying Zarr archive to the Hub. This may take a while.")
zarr.copy_store(
source=dataset.zarr_root.store.store,
dest=dest.store,
log=print,
if_exists="skip",
)
dest = client.open_zarr_file(
File "/novo/users/fegt/miniconda3/envs/bend/lib/python3.10/site-packages/polaris/hub/client.py", line 333, in open_zarr_file
raise PolarisHubError("Error opening Zarr store") from e
polaris.utils.errors.PolarisHubError: Error opening Zarr store
But can also just wait until there is a retry mechanism for now.
Thanks! #144 will also add more informative error messages for this case, helping us to investigate.
I added myself temporarily to the mlls
organization and I see the issue! It's because of mode="w"
. From the Zarr docs:
‘w’ means create (overwrite if exists);
Since we haven't implemented the delete operation, overwriting the archive fails.
This would be fixed by changing it to mode="a"
.
Polaris version
dev
Python Version
3.10
Operating System
Linux
Installation
pip
Description
I'm trying to upload a zarr dataset. Not doing anything special as far as I can tell - I think the zarr upload fails, but the dataset gets created on the hub anyway. Not sure what's going wrong
Steps to reproduce
Additional output
No response