FileNotFoundError on establish command

[Alireza-Farahani]

I have a typical scrapy project as seen below but when I execute testmaster establish <spider name>, inside 'NewsCrawler' folder, I get following errors:

  File "/home/alireza/PycharmProjects/NewsCrawler/venv/lib/python3.8/site-packages/scrapy_testmaster/cli.py", line 274, in parse_command
    self.establish()
  File "/home/alireza/PycharmProjects/NewsCrawler/venv/lib/python3.8/site-packages/scrapy_testmaster/cli.py", line 250, in establish
    for callback in get_callbacks(spider_path):
  File "/home/alireza/PycharmProjects/NewsCrawler/venv/lib/python3.8/site-packages/scrapy_testmaster/utils_novel.py", line 266, in get_callbacks
    with open(spider_path, 'r') as spider_file:
FileNotFoundError: [Errno 2] No such file or directory: '/home/alireza/PycharmProjects/NewsCrawler/NewsCrawler/spiders/<spider name>.py'

As seen in last line, root folder is repeated 2 times which cause the problem.

[ThomasAitken]

Thanks for submitting your issue and sorry to not get back immediately.

I have now published a new version of the library (0.1.7) which resolves this issue.

This library had previously assumed that the user has created a project whose name is the same as the outer folder, i.e. they have used "scrapy startproject myproject" without the extra arg "[project_dir]" with which one can specify the name of the outer folder separately from the inner project. This is now fixed.

Resolving this issue has given me extra reason to think that it might be better to place the tests within the actual project folder rather than just inside the outer folder (i.e. "news_crawler/tests/spider_name/callback_name" rather than "NewsCrawler/testmaster/tests/..."). With the current structuring, if you had two spiders with the same name in different projects within the NewsCrawler directory, this library would treat them as the same spider.

I may implement this change to the directory structure if I have time in the coming weeks.