search

podaac / data-subscriber

Subscribe and bulk download collections of data at PO.DAAC
Apache License 2.0
83 stars 29 forks source link

Issues/91 #92

Closed mike-gangl closed 2 years ago

mike-gangl commented 2 years ago

Added 'citation' text file after subscription and downloader runs.

frankinspace commented 2 years ago

Looks like some of the regression tests need to be updated to take into account an additional file potentially being created in the destination directory.

mike-gangl commented 2 years ago

@frankinspace agreed, i should have updated those before the PR. got caught up on something but will make them in a minute.

frankinspace commented 2 years ago

@mike-gangl I enabled debug logging during regression tests so we can see the logs when there are failures, it looks like it is still random 401 errors from CMR causing the problem.

I also set the max parallel to 2 so it will only run 2 combinations of the matrix in parallel in the hopes that reducing the number of parallel tests we'd get some more consistency (at the expense of the tests taking longer). It looks like that did help a bit but is still not enough.

mike-gangl commented 2 years ago

thanks @frankinspace . I've updated the test to look for the data files only given that a citation file might not be created. i figured it wouldn't be created due to a different DAAC not having the required information, but token issues in CMR are another reason here.