Closed KrzysztofMadejski closed 7 years ago
Instead of checking site_url, why not check if the resource has type='upload'
? Otherwise you'll archive resources that are "uploaded" but aren't living on the server like those uploaded by ckanext-cloudstorage
.
@amercader @davidread Any suggestions where this should be coded?
I guess either in a) https://github.com/ckan/ckanext-archiver/blob/master/ckanext/archiver/tasks.py#L232 ("Won't attemp to download")
or in b) download
function https://github.com/ckan/ckanext-archiver/blob/master/ckanext/archiver/tasks.py#L338 (checking type=upload
as @TkTech suggested) (throwing ChooseNotToDownload?)
I'd go for a) and prefilling proper download
function response using existing file:
return {'mimetype': mimetype,
'size': length,
'hash': hash,
'headers': dict(res.headers),
'saved_file': saved_file_path,
'url_redirected_to': url_redirected_to,
'request_type': method}
Regarding @TkTech 's suggestion, if the file is in the cloud then ckanext-qa can't run a hash on it, or libmagic etc - you need a local copy for that, so I'd suggest sticking with your original plan, or adding a config option.
I'd start down the path of a) and see if it works out.
@davidread what does ckanext-qa need? ckanext-cloudstorage
can populate the mimetype and hash fields which are core fields on a resource as it's uploaded.
Having large resources (say >=4GB) uploaded to your s3/azure/whatever bucket, then downloaded locally again kind of defeats the point. You need to be able to disable this for things you're already hosting.
@TkTech clearly you can optimize for whichever situation - I'm not saying you're wrong. But ckanext-archiver is designed for providing ckanext-qa and ckanext-packagezip and the like with local file access. If you want a solution optimized for simply caching files except the ones like these, then I suggest you fork it.
I've also understood that the main goal of archiver was caching local externally hosted files, which availability "we as dataportal provider" cannot assure. Coding qa
as dependent on archiver is a wise solution, but I see that also other providers could offer qa the info it needs, though it's not trivial. Some feaures (like hash and libmagic) would need to be run on the storage server.
@TkTech I will start another issue for discussion on how ckanext-cloudstorage
and ckanext-qa
could work together.
That was my understanding as well. I want ckanext-archiver
and ckanext-qa
to run only on files our portal does not already have complete control over (type='upload'
), those being externally hosted. This seems like a common enough case and simple enough to do just as an ini setting unless I'm missing something?
A simple config setting for this is fine.
QA does various things too. It would make little sense to exclude some files from the 5 stars of openness report, for example. But if the bits of Archiver and QA you do want to use don't make sense to you for some files then having a simple config option for that is fine.
My latest archiver: https://github.com/ckan/ckanext-archiver/commit/5668c5377351e93babe0fa2ea3d38cfd4d693f38 + #33 PR archives resources that were uploaded to server. It's a waste of space as these are files are already stored on the server.
Proposed solution:
ckan.site_url
ckan.storage_path
and a remote url when the resource is served by ckan.