kyleam commented 5 years ago

These tests no longer fail in V6 mode with the current DataLad (v0.11.0) and its minimum git-annex version (6.20180913).

The only remaining usage of known_failure_v6 is in test_openfmri.py. Under V6, these fail with

  File "/home/kyle/src/python/datalad-crawler/datalad_crawler/dbs/base.py", line 86, in save
    with open(self._filepath, 'w') as f:
IOError: [Errno 13] Permission denied: '[...]/.datalad/crawl/statuses/incoming.json'

suggesting that the crawler should set git attributes so that that file (and perhaps others) is added to git rather than the annex.

yarikoptic commented 5 years ago

I hate that crappy bug with scrapy leading to TypeError: attrib() got an unexpected keyword argument 'converter' - I never found a solid solution. restarted that run for now with hope for the better luck

kyleam commented 5 years ago

restarted that run for now with hope for the better luck

Thanks. Passed now.

yarikoptic commented 5 years ago

Apparently there is no v6 in Travis, so we shouldn't drop them until we verify that it actually works in v6. Indeed it needs .gitattribute setting to be done in Annexificator to avoid those status files (not openfmri specific) to be set

kyleam commented 5 years ago

Apparently there is no v6 in Travis

Eh, thanks for spotting that. I just assumed there was a v6.

so we shouldn't drop them until we verify that it actually works in v6.

Actually works in v6 on Travis, that is. I verified locally before creating this commit, of course.

Do you want to extend the matrix to include a V6 run?

kyleam commented 5 years ago

Do you want to extend the matrix to include a V6 run?

Or perhaps replace the direct mode run with a V6 one?

yarikoptic commented 5 years ago

Since we didn't rip out direct in released main code, I be would prefer to keep it

codecov[bot] commented 5 years ago

Codecov Report

Merging #23 into master will decrease coverage by 0.01%. The diff coverage is 100%.

@@            Coverage Diff             @@
##           master      #23      +/-   ##
==========================================
- Coverage   86.45%   86.44%   -0.02%     
==========================================
  Files          51       51              
  Lines        4135     4130       -5     
==========================================
- Hits         3575     3570       -5     
  Misses        560      560

Impacted Files	Coverage Δ
datalad_crawler/pipelines/tests/test_openfmri.py	`91.35% <ø> (-0.04%)`	:arrow_down:
datalad_crawler/pipelines/tests/test_balsa.py	`100% <ø> (ø)`	:arrow_up:
datalad_crawler/tests/test_crawl_init.py	`100% <100%> (ø)`	:arrow_up:
datalad_crawler/nodes/tests/test_annex.py	`99.58% <100%> (-0.01%)`	:arrow_down:
...awler/pipelines/tests/test_simple_with_archives.py	`100% <100%> (ø)`	:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update e7f192a...41b9af4. Read the comment docs.

kyleam commented 5 years ago

so we shouldn't drop them until we verify that it actually works in v6.

It looks like the ones that were dropped pass in V6, but there is still an additional V6 failure. We should mark test_simple_with_archives.test_simple1 as a known failure due to the same "permission denied" issue above.

https://travis-ci.org/datalad/datalad-crawler/jobs/450911228

datalad / datalad-crawler

TST: Drop stale known_failure_v6's #23

Codecov Report