Closed FuhuXia closed 3 weeks ago
A PR is submitted to upstream https://github.com/ckan/ckanext-spatial/pull/337
But we want to stay at current release version and cherry-pick this fix. So we should use our fork in catalog repo's requirement.in.
Still have issue with harvest source https://hazards.fema.gov/filedownload/metadata/, the harvester does not traverse into folders. It turns out harvester is expecting relative path for each folder, for example, for a WAF like this
Sunday, April 14, 2024 7:55 PM <dir> R01
Sunday, April 14, 2024 7:55 PM <dir> R02
Harvester is expecting R01/
, R02/
, but this IIS WAF is using full path /filedownload/metadata/R01/
, /filedownload/metadata/R02/
. Harvester is designed to ignore any path starting with /
, as in this code.
Nginx and Apache servers are fine. We need to research if IIS is using full path by default, or it is a custom setting by this particular IIS server, and then come up with a fix accordingly.
Further fix is done to address IIS folder url.
Fail to harvest source https://hazards.fema.gov/filedownload/metadata/. Something to do with https://github.com/ckan/ckanext-spatial/issues/319. Need to exam the iis parser and make it more inclusive.
How to reproduce
harvest IIS WAF with subfolders like this
Expected behavior
traverse into
mydir
and harvest files under the sub folderActual behavior
ignore files under
mydir
Sketch
[Notes or a checklist reflecting our understanding of the selected approach]