search

demiangomez / Parallel.GAMIT

Python wrapper to parallelize GAMIT executions
BSD 3-Clause "New" or "Revised" License
36 stars 17 forks source link

pyArchiveService.py behavior #10

Closed smalleyr closed 6 years ago

smalleyr commented 6 years ago

Started with ~1100 files missing from an igs continuous station (got our attention because entry in one of ? directories, but no rinex in the repository tree, no locks. Got the 1100 files by looking at data in osu archive and moving the ones not in the PG archive into data_in).

Ran pyArchiveSevice.py on these 1100 files. About half were moved to the archive (or at least disappeared from the repository directory tree. A handful ended up in data_retry_in and data_rejected, the remaining ones still in data_in (the number of files in the archive grew by 500).

There were no locked files and probably a single error in the error message file (errors_pyArchiveService.log in /Volumes/UsersDrive/Users/smalley/Working.Parallel.GAMIT/run_dir on capybara). I think each run of pyArchiveService.py generates one error.

Re-running pyArchiveService.sh started reporting 120 files (ls | wc shows ~575, and the number of files in the archive is constant), and each iteration it drops the number of files reorted by 5-7 files (a few times it dropped by more). I think it added one line to the error file each run).(see screen output).

So - files in data_in not going anywhere (from ls), but "disappearing" from processing by apArchiveService.py, not locked and no errors.

demiangomez commented 6 years ago

pyArchiveService only takes d.Z files, it doesn't recognize o.Z files, which are not RINEX-style compliant.