search

aria-jpl / scihub_acquisition_scraper

Apache License 2.0
0 stars 5 forks source link

Standard-product acquisitions for hysds3 #10

Closed shitong01 closed 4 years ago

shitong01 commented 4 years ago

Standard-product acquisitions for hysds3

~1. revert changes to acquisitions naming~

  1. correct command used for ipf_scraper
  2. Fix encoding issue with ipf_scraper_scihub: 
    Failed to get IPF for acquisition-Sentinel-1A_20200316T110726.590Z_128_IW-esa_scihub. 
cannot use a string pattern on a bytes-like object. 
Traceback (most recent call last):
File "/home/ops/verdi/ops/scihub_acquisition_scraper/ipf_scrape/ipf_version.py", line 215, in 
ipf = extract_scihub_ipf(met)
File "/home/ops/verdi/ops/scihub_acquisition_scraper/ipf_scrape/ipf_version.py", line 180, in extract_scihub_ipf
ipf = get_scihub_ipf(manifest)
File "/home/ops/verdi/ops/scihub_acquisition_scraper/ipf_scrape/ipf_version.py", line 90, in get_scihub_ipf
ns = get_scihub_namespaces(manifest)
File "/home/ops/verdi/ops/scihub_acquisition_scraper/ipf_scrape/ipf_version.py", line 80, in get_scihub_namespaces
matches = re.findall(r'\s+xmlns:?(\w*?)\s*=\s*[\'"](.*?)[\'"]', xml)
File "/home/ops/verdi/lib/python3.7/re.py", line 223, in findall
return _compile(pattern, flags).findall(string)
TypeError: cannot use a string pattern on a bytes-like object

crontabs:

# scihub acquisition scrapers
15 * * * * source $HOME/.bash_profile && $HOME/verdi/bin/python /home/ops/verdi/ops/scihub_acquisition_scraper/crons/acq_ingest_cron.py --tag develop-standard-product  --dataset_version v2.0 --hours 6 opensearch  > $HOME/verdi/log/scihub_acquisition_scraper_hour.log 2>&1
45 14 * * * source $HOME/.bash_profile && $HOME/verdi/bin/python /home/ops/verdi/ops/scihub_acquisition_scraper/crons/acq_ingest_cron.py --tag develop-standard-product  --dataset_version v2.0 --days 5 opensearch  > $HOME/verdi/log/scihub_acquisition_scraper_day.log 2>&1

# ipf scrapers
57 */6 * * * source $HOME/.bash_profile && $HOME/verdi/bin/python $HOME/verdi/ops/scihub_acquisition_scraper/crons/ipf_global_cron.py --tag=develop-standard-product > $HOME/verdi/log/cron-scrape_global_ipf.log 2>&1

logs: successful acquisition scraping successful ipf scraping

shitong01 commented 4 years ago

Do not merge yet, until I post the logs in the PRs.

shitong01 commented 4 years ago

@pymonger This is ready for merging / another review, added logs and verified working.

pymonger commented 4 years ago

LGTM. @torresal, @mlucas2018: can you also verify?