Closed KrzysztofMadejski closed 7 years ago
TkTech
commented
7 years ago Instead of checking site_url, why not check if the resource has type='upload'? Otherwise you'll archive resources that are "uploaded" but aren't living on the server like those uploaded by ckanext-cloudstorage.
KrzysztofMadejski
commented
7 years ago @amercader @davidread Any suggestions where this should be coded?
I guess either in a) https://github.com/ckan/ckanext-archiver/blob/master/ckanext/archiver/tasks.py#L232 ("Won't attemp to download")
or in b) download function https://github.com/ckan/ckanext-archiver/blob/master/ckanext/archiver/tasks.py#L338 (checking type=upload as @TkTech suggested) (throwing ChooseNotToDownload?)
I'd go for a) and prefilling proper download function response using existing file:
return {'mimetype': mimetype,
'size': length,
'hash': hash,
'headers': dict(res.headers),
'saved_file': saved_file_path,
'url_redirected_to': url_redirected_to,
'request_type': method}
davidread
commented
7 years ago Regarding @TkTech 's suggestion, if the file is in the cloud then ckanext-qa can't run a hash on it, or libmagic etc - you need a local copy for that, so I'd suggest sticking with your original plan, or adding a config option.
I'd start down the path of a) and see if it works out.
TkTech
commented
7 years ago @davidread what does ckanext-qa need? ckanext-cloudstorage can populate the mimetype and hash fields which are core fields on a resource as it's uploaded.
Having large resources (say >=4GB) uploaded to your s3/azure/whatever bucket, then downloaded locally again kind of defeats the point. You need to be able to disable this for things you're already hosting.
davidread
commented
7 years ago @TkTech clearly you can optimize for whichever situation - I'm not saying you're wrong. But ckanext-archiver is designed for providing ckanext-qa and ckanext-packagezip and the like with local file access. If you want a solution optimized for simply caching files except the ones like these, then I suggest you fork it.
KrzysztofMadejski
commented
7 years ago I've also understood that the main goal of archiver was caching local externally hosted files, which availability "we as dataportal provider" cannot assure. Coding qa as dependent on archiver is a wise solution, but I see that also other providers could offer qa the info it needs, though it's not trivial. Some feaures (like hash and libmagic) would need to be run on the storage server.
@TkTech I will start another issue for discussion on how ckanext-cloudstorage and ckanext-qa could work together.
TkTech
commented
7 years ago That was my understanding as well. I want ckanext-archiver and ckanext-qa to run only on files our portal does not already have complete control over (type='upload'), those being externally hosted. This seems like a common enough case and simple enough to do just as an ini setting unless I'm missing something?
davidread
commented
7 years ago A simple config setting for this is fine.
QA does various things too. It would make little sense to exclude some files from the 5 stars of openness report, for example. But if the bits of Archiver and QA you do want to use don't make sense to you for some files then having a simple config option for that is fine.
My latest archiver: https://github.com/ckan/ckanext-archiver/commit/5668c5377351e93babe0fa2ea3d38cfd4d693f38 + #33 PR archives resources that were uploaded to server. It's a waste of space as these are files are already stored on the server.
Proposed solution:
ckan.site_urlckan.storage_pathand a remote url when the resource is served by ckan.