[QA] Jackett test automation

[ngosang]

@ngosang

I tested all public trackers and they are in good shape. I would like to test semi and private trackers too but I only have access to few of them. I have several test automated to test all configured trackers with several searches. Do you have a Jackett hosted somewhere so I can launch the tests in the private trackers? If not, I can give you the automated tests, but they need some work to be usable. We can talk in private if you prefer ngosang [@ hotmail.es

@garfield69

i'm in the process of adding sort order options to the indexer configs of any sites I have access to. many of the rest may never get updated until some one raises a ticket and can provide access. So I'll refresh those with my pool and test and check the logs for errors as I go.

@ngosang

IMHO, you are supporting 400 sites that change a lot. The only way to keep this up to date is automation.

1. Have a Jackett instance running 24x7 in some server. This Jackett should have all trackers configured and auto-update enabled. This will test that Jackett updates are working and will be used in the manual/automated tests. If this project have several collaborators, doesn't make sense that everybody has 300 accounts... a shared server makes sense. I can host it.
2. Automatic tests. The automatic tests will be launched everyday in the 24x7 Jackett. For each tracker it will test: 1 word search, 2 word search, uppercase, reverse order, bad search, result fields, backup domains, sort order... The test will be in some git repository and can be updated anytime.
3. Reporting. The automatic tests will generate a report with all the results. Can be an html with colors, email errors... When there are errors you will see clearly the tracker, search term, number of results and exception. Can be automated to open issues in Github if necessary. Think about this, it will save a lot of time.

@garfield69

what you are proposing sounds wonderful, and if you can provide a server then I applaud your commitment to helping out with Jackett. Testing for the common site-has-changed-domain would particularly be useful. I do have one reservation however: do not include private indexers. reasons: public and semi-private are open and if logins on semi-private site get suspended due to errors its no big deal. But private sites are more fussy about violations, and can suspend or even delete your account. Many private sites use cookies which expire and require the indexer configs to be reloaded with new cookies at regular intervals. Not the kind of thing that can be recovered from with automation perhaps. Many private sites auto-redirect you to Notices or unread mail pages after login which Jackett treats as errors. And finally, in these days of extreme malicious hacking, having profiles with private login data on a server makes me uncomfortable. Stories are published daily about data theft and I would not want anyone's private-site logins made public by accident or intent.

[ngosang]

I do have one reservation however: do not include private indexers.

I can test the public trackers in my pc, I don't need help for that. The point is to share some private tracker accounts to test all the trackers. I understand your concerns but:

• It will be private, hosted in my server and password protected. Only you and I will have access (I don't know the other collaborators).
• I'm not asking to share your personal accounts. We should create new ones with fake emails. I have invitations for several trackers and I'm sure I can get more for the cause.
• I don't think the cookies and terms violations will be a problem. The server have static IP, the cookie will be configured by hand and updated when it expires. We shouldn't use that accounts for download or personal use.

We can start only with public trackers and add private trackers eventually. Let me thing about that this weekend.