Closed matthewjmckinley closed 2 years ago
I've tried doing the Dataverse one a couple times and keep getting errors, here's the one I did from the most basic url to make sure I was understanding correctly: https://dataverse.dartmouth.edu/dataverse/healthefficiency
2022-05-16T23:36:07+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omekavO8CoJ.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:07+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omekal40xsW[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omekavO8CoJ.jpg'" failed with status code 1.
2022-05-16T23:36:08+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omekabce6fn.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:08+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omekaXXfa0x[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omekabce6fn.jpg'" failed with status code 1.
2022-05-16T23:36:11+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omekad3ASyU.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:11+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omeka3vB8hW[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omekad3ASyU.jpg'" failed with status code 1.
2022-05-16T23:36:12+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omeka5gcWFU.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:12+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omekaf03GjU[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omeka5gcWFU.jpg'" failed with status code 1.
2022-05-16T23:36:15+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omekaJqx5ho.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:15+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omekat5CQ6e[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omekaJqx5ho.jpg'" failed with status code 1.
2022-05-16T23:36:15+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omekaV2EzgH.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:15+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omekab8pFux[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omekaV2EzgH.jpg'" failed with status code 1.
2022-05-16T23:36:16+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omekadgQmE4.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:16+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omeka32h9BS[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omekadgQmE4.jpg'" failed with status code 1.
2022-05-16T23:36:16+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omeka3LD3nt.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:16+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omekaPRK9Hg[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omeka3LD3nt.jpg'" failed with status code 1.
2022-05-16T23:36:17+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omekan6LBtT.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:17+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omekaX1mx5F[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omekan6LBtT.jpg'" failed with status code 1.
2022-05-16T23:36:17+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omeka5jwfTk.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:17+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omekanK0lT6[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omeka5jwfTk.jpg'" failed with status code 1.
2022-05-16T23:36:17+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omekaxT6UvQ.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:17+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omekaR9lHpA[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omekaxT6UvQ.jpg'" failed with status code 1.
2022-05-16T23:36:18+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omekaXnWkrn.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:18+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omeka5N2tD6[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omekaXnWkrn.jpg'" failed with status code 1.
2022-05-16T23:36:18+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omekaxsgUJV.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:18+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omekaPgYygE[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omekaxsgUJV.jpg'" failed with status code 1.
2022-05-16T23:36:18+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omekaR2XXov.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:18+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omeka5fJxgd[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omekaR2XXov.jpg'" failed with status code 1.
2022-05-16T23:36:18+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omekazJtxk6.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:18+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omekaFSYTyN[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omekazJtxk6.jpg'" failed with status code 1.
2022-05-16T23:36:23+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omekaDYK01F.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:23+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omekatFk11a[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omekaDYK01F.jpg'" failed with status code 1.
2022-05-16T23:36:24+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omekar9pC0g.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:24+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omekaxMaruJ[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omekar9pC0g.jpg'" failed with status code 1.
2022-05-16T23:36:24+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omekaHcTTZU.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:24+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omekaj0jBtm[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omekaHcTTZU.jpg'" failed with status code 1.
2022-05-16T23:36:25+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omekajKh7O2.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:25+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omekajvfKxt[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omekajKh7O2.jpg'" failed with status code 1.
2022-05-16T23:36:25+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omekazGbx2b.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:25+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omekaNHpT8B[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omekazGbx2b.jpg'" failed with status code 1.
2022-05-16T23:36:25+00:00 ERR (3): convert: no decode delegate for this image format ' @ error/constitute.c/ReadImage/564. convert: no images defined
/tmp/omekaL5UPWW.jpg' @ error/convert.c/ConvertImageCommand/3235.
2022-05-16T23:36:25+00:00 ERR (3): Command "/usr/bin/convert '/tmp/omekadO7ram[0]' -auto-orient -background white +repage -alpha remove -thumbnail '800x800>' '/tmp/omekaL5UPWW.jpg'" failed with status code 1. 2022-05-16T23:36:25+00:00 ERR (3): exception 'Omeka\Api\Exception\ValidationException' in /var/www/html/katknow/OmekaSTesting/application/src/Api/Adapter/AbstractEntityAdapter.php:649 Errors: { "o:media": [ { "file": [ "Error validating \"\/api\/access\/datafile\/365\". Cannot store files with the media type \"text\/x-python\".", "Error validating \"\/api\/access\/datafile\/365\". Cannot store files with the resolved extension \"\"." ] }, { "file": [ "Error validating \"\/api\/access\/datafile\/369\". Cannot store files with the media type \"text\/x-python\".", "Error validating \"\/api\/access\/datafile\/369\". Cannot store files with the resolved extension \"\"." ] }, { "file": [ "Error validating \"\/api\/access\/datafile\/371\". Cannot store files with the media type \"text\/x-python\".", "Error validating \"\/api\/access\/datafile\/371\". Cannot store files with the resolved extension \"\"." ] } ] } Stack trace:
This is more of a thumbnail conversion issue than an issue with the Data Repository Connector code--as I mentioned in the testing outline, data repositories can contain a lot of less standard file extensions that are not in Omeka's approved list, resulting in errors such as this when trying to convert files.
You'll have to either add them manually to the types/extension list in Settings as you encounter them, or try to find another collection with more common file types. It's a pain, I know.
Oh okay thanks I didn't realize it would derail the whole thing, I thought it would let the pdfs etc. through still. But that makes sense--thanks!
I'm not 100% sure, but is this error for Zenodo regarding something else:
Fatal error: Uncaught Error: Class 'DataRepositoryConnector\DataRepoSelector\Exception\RuntimeException' not found in /var/www/html/katknow/OmekaSTesting/modules/DataRepositoryConnector/src/DataRepoSelector/Zenodo.php:78 Stack trace:
in /var/www/html/katknow/OmekaSTesting/modules/DataRepositoryConnector/src/DataRepoSelector/Zenodo.php on line 78
I tried re-running it and it may be working now so we'll see
EDIT: Okay, so I tried again and it didn't work, then I realized I didn't include the slash so tried again with slash and then got this error:
2022-05-17T13:36:24+00:00 ERR (3): Laminas\Http\Client\Adapter\Exception\TimeoutException: Read timed out after 10 seconds in /var/www/html/katknow/OmekaSTesting/vendor/laminas/laminas-http/src/Client/Adapter/Socket.php:690 Stack trace:
Like I said, it looks different so I don't think it's the same as the one where I need to edit the file types but apologies if it is...I don't always understand what I'm looking at with the errors!
I made a few new commits that should make the error reporting more descriptive & clarifies the helper text for the Zenodo Community ID. Basically the string should only be something like lory_hslu
without all the search parameters that follow, but this was unclear in the previous helper text.
I'm testing right now on that collection and also getting a Timeout
error eventually. I've run into this with the DSpace Connector too, and a lot of times it has to do with the remote server getting too many requests at once. It's not really something we can properly fix on our end since the issue is happening on their end. I've had success with DSpace by knocking the limit down, so I'm currently re-running with lory_hslu
and a limit of 10
to see if that helps.
I've been playing with this on and off all day, and it seems Dataverse works on all points using the dcterms. Was able to import, undo, large numbers, etc. Could you give me more information on the other metadata schemes? I think I imported schema.org correctly, so would I just need to find a dataverse that uses that kind of metadata? Also, could you explain how to install the oai_dc? I just can't figure that one out for some reason. Thanks!
It doesn't matter what the original metadata format is within the Dataverse instance--the export will convert it to whatever format you choose. You would then just see if the values passed through correctly. You might lose some fields/values if, for example, there's a dc:terms field that doesn't have a schema.org equivalent.
For oai_dc, you should be able to use http://purl.org/dc/elements/1.1/ as the namespace and one of the files under that URI's heading here for the vocab file: https://www.dublincore.org/schemas/rdfs/
oai_dc for dataverse is coming over nicely, except the title while being imported as part of the item is not displaying at the top of the item page/in the listing of items:
Right, the display title is hardcoded into the Omeka S code as dcterms:title, meaning the page title and search/browse results will only display if there is a dcterms:title metadata value associated with the item. I don't think there's currently a way to customize this.
Sounds good--if that's what's going on then the oai_dc is working for both Dataverse & Zenodo. dcterms works great with Dataverse as well. I also tried schema.org, which looks like this, but that may just be an indicator I added it incorrectly (although it does show that it's in my instance):
At the very least, the link it provides does work.
I'm having some timeout issues with CKAN, but I'm going to keep tweaking it down until it works.
I'm down to 10 with CKAN and it's not working, but this message looks a little different so I don't think this one is timing out but I'm not 100% sure:
2022-05-23T18:13:29+00:00 ERR (3): Omeka\Job\Exception\RuntimeException: Requested "https://data.humdata.org" got "HTTP/1.1 409 Conflict". in /var/www/html/katknow/OmekaSTesting/modules/DataRepositoryConnector/src/DataRepoSelector/CKAN.php:77 Stack trace:
Not very familiar with that particular error, but I suspect it may still be an issue of overloading the server, i.e. sending too many requests and one of them got garbled and caused a conflict error. These sort of errors can be common when trying to download larger record sets at once (as is the case for data.humdata.org, which appears to have almost 20,000 records).
I wonder if manually upping the timeout threshold, in the same way we do for the DSpace Connector, might help here. In the omeka-s/modules/DataRepositoryConnector/src/Job/Import.php
file, could you try adding the following line:
$this->client->setOptions(['timeout' => 120]);
directly underneath line 34 where the client is initially set with $this->client = $this->getServiceLocator()->get('Omeka\HttpDClient');
, saving the file and retrying?
You should be able to start with a larger limit like the default 100 before bringing it down again if it keeps encountering issues, but this should hopefully help cut down on timeout errors.
That change worked like a charm--limit was set at 100, but it imported the full 246 in a matter of minutes. Other than the weird view on schema.org metadata, I think this works really well now. One thing I did notice however is in the descriptor text for limit on the CKAN page it still says "Zenodo."
Great! I've committed that timeout threshold change. And good catch on the form error, fixed in another commit.
For the schema.org issue, I worked out that it was due to a small thing--the prefix associated with schema.org in your vocabulary was "schema.org" when it should just be "schema". I deleted and reimported the vocabulary with this change and the schema.org metadata is now showing: http://dev.omeka.org/katknow/OmekaSTesting/admin/item/31407
Ahhhh oops. Yeah, now that I know that about schema it's working great, thanks! I think this is good to go. Only errors I'm ever getting now is the thumbnail/filetype error.
This module will allow users to import records and/or data files from several different data repository platforms: Dataverse, Zenodo & CKAN.
Dataverse: you can find many Dataverse instances to test with at the bottom of this page: https://dataverse.org/metrics . You can usually find sub-'Dataverses' under the main instance from the search/browse page. The 'Dataverse Identifier' field in the module will be the string after '/dataverse/' in the sub-dataverse URL. Zenodo: Zenodo is one large shared repository; you can find the sub-'Communities' here: https://zenodo.org/communities/ . Again, the 'Zenodo Community ID' is found at the end of a communities URL CKAN: Used for large consortium, usually governmental data collections. You can find some examples here: https://ckan.org . I've been testing with US and Canada but feel free to try others. On most instances, clicking the search button with no input will return all records, then 'CKAN Organizations' can be found in the left sidebar. Once in an organization, the value you want follows 'organization=' in the URL
Testing Tasks
For each service