behinger
commented
2 weeks ago any update on this? I'm happy to spend ressources to try to debug this, but some tipps on what to look for would be nice.
Closed behinger closed 19 hours ago
behinger
commented
2 weeks ago any update on this? I'm happy to spend ressources to try to debug this, but some tipps on what to look for would be nice.
shoeffner
commented
4 days ago I was able to reproduce this issue, find the root cause of this, and patch around it.
Root cause:
During upload, dataverse converts various tabular data (R, csv, and more) to a specific tab-separated tab file. See also https://guides.dataverse.org/en/6.2/api/dataaccess.html#all-formats-bundled-download-for-tabular-files, and the very beginning of the guides:
By default, tabular files are downloaded in their “archival” form (tab-separated values). To download the original files (Stata, for example), add
format=originalas a query parameter.
In the case of your tsv files, they will probably only differ slightly, but for csv files, the download differs greatly. Either way, the hashes will not match between the uploaded and downloaded file.
So the solution is to set the format parameter. I believe for datalad-dataverse it should almost always be original, so we can change the call in dataset.py:
From e611697f466e605d8b21505a085de18380f109e6 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Sebastian=20H=C3=B6ffner?= <info@sebastian-hoeffner.de>
Date: Tue, 16 Jul 2024 03:46:22 +0200
Subject: [PATCH] Always use the original file format when downloading files.
Otherwise, the files will not have the correct hashes, as dataverse replaces, e.g., csv or R table files with its own tabular data format.
---
datalad_dataverse/dataset.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/datalad_dataverse/dataset.py b/datalad_dataverse/dataset.py
index 7fb8f52..94d7d74 100644
--- a/datalad_dataverse/dataset.py
+++ b/datalad_dataverse/dataset.py
@@ -171,7 +171,7 @@ class OnlineDataverseDataset:
# https://github.com/gdcc/pyDataverse/issues/49
# the code below is nevertheless readied for such a
# scenario
- response = self.data_access_api.get_datafile(fid)
+ response = self.data_access_api.get_datafile(fid, data_format="original")
# http error handling
response.raise_for_status()
with path.open("wb") as f:
--
2.45.2
I can create a proper PR and the reproduction steps tomorrow, if you want. Until then, @behinger, maybe you want to fix your local copy with that patch and see if it solves your problem?
While I was working on this issue, I noticed a few other minor issues: the demo.dataverse.org instance did not work with is_pid for me, I had to manually set it to False; and httpx does not support iter_content(...) but instead iter_bytes(...), plus the requests property ok on a response object is missing. I haven't looked at the history, but I guess this is because I used the main branch and not some release. Should I create follow-up issues?
adswa
commented
4 days ago sorry all for the lack of responses and time, and many many thanks for the analysis and patch! We've been struggling how to handle dataverse API changes in recent versions while maintaining compatibility with previous APIs (e.g., our own institution's dataverse is still on v4.x, and the format parameter seems to have been introduced in v5.x.). I'm adding this issue to our weekly meeting for a renewed discussion how to approach this best. Thanks both for creating momentum here. A PR and issues would be most welcome, though, the current state is clearly not ideal.
shoeffner
commented
3 days ago I created a couple of PRs now to fix some of the issues, however, some of that might not be compatible with dataverse 4.x anymore, maybe not even with 5.x. I am happy to change the PRs to try to be compatible with all versions, 4, 5, and 6. Unfortunately, the demo instance uses dataverse 6 and the docker containers seem to only offer a version 6 as well, so it would require quite some setup to test it against older instances. It's unfortunate that even though dataverse offers a versioned API, they do not seem to have made much use of it, not even for the documented breaking changes.
I am looking forward for the outcome of your weekly!
Hi! we are trying to use this bridge. Pushing & cloning works fine, but I get a strange error I cannot pinpoint.
I already followed some debugging instructions and now instead of
datalad get, I run:❯ git annex get sub-001_ses-001_scans.tsv -dThis ultimately leads to the following error
The full output is below.
I did manually check the md5sum because initially I thought they surely will be different because of tabular-ingest or something. But no, the md5 online, after download and the hash in the filename are the same.
Now I'm out of ideas, and maybe someone can help me figure this out.
Best, Bene
PS: I updated today datalad + the plugin.
EDIT: Only CSV and TSV files are affected, e.g. JSON works fine.
Full log: