search

scrapy / scrapyd

A service daemon to run Scrapy spiders
https://scrapyd.readthedocs.io/en/stable/
BSD 3-Clause "New" or "Revised" License
2.96k stars 569 forks source link

Version 1.3 #364

Closed sluceno closed 2 years ago

sluceno commented 4 years ago

Hi there,

is it possible to bump a 1.3 version? It would be good to have a version with support for Basic Authentication #326

Thanks

my8100 commented 4 years ago

For now, simply execute:

pip install -U git+https://github.com/scrapy/scrapyd.git
sluceno commented 4 years ago

Yes sure, this is what we are doing right now

Do you have a rough estimate of when 1.3 will be released?

Digenis commented 3 years ago

Hi @sluceno

See #389

If you tried the master branch, any feedback is highly appreciated.

I believe @my8100 will soon merge #337. Make sure to check if this changes anything for your setup.

my8100 commented 3 years ago

@Digenis

337 merged.

Seems we have to update the code for Travis CI: https://travis-ci.org/github/scrapy/scrapyd/builds/766655073 Also, any plan to drop support for Python 2.7 and other old versions?

Digenis commented 3 years ago

Cc: @sluceno

Edit: any feedback is welcome. E.g. something along the lines:

scrapyd git version 788ee1b98006721103115678c0ac6ded769675b1 scrapy so-and-so version twisted so-and-so version python so-and-so version Works as expected

Digenis commented 3 years ago

@my8100, yes.

For 1.3, we can keep it as is. The only issue I see is bracketing package requirements' versions properly. Especially twisted and scrapy.

In scrapyd-1.4 we'll definitely drop py2.7 and let's also drop py3.4 and py3.5. Maybe we can also make the code more python3-idiomatic, especially if we drop py3.6.

I had happily forgotten about the cryptography update. Using binary wheels from pypi should be enough for the CI.

my8100 commented 3 years ago

Cc: @sluceno

Edit: any feedback is welcome. E.g. something along the lines:

scrapyd git version 788ee1b scrapy so-and-so version twisted so-and-so version python so-and-so version Works as expected

@Digenis I have done some basic test for the latest code on Windows/Ubuntu, Python2.7/3.8, with basic auth enabled/disabled this weekend. I think you had better try it out before releasing the new version.

Digenis commented 3 years ago

As I said in the other issue, I can't do large scale testing anymore but I'll add some more environments to tox and bracket requirements' versions.

For alpha and beta releases that would be enough.

I fixed the cryptography issue.

There's another problem in test_failed_spider_list where a deprecation warning gets in the way of parsing a traceback from the subprocess's stdout.

I'll try mocking Popen with a wrapper that adds python arguments that disable warnings.

Digenis commented 3 years ago

There are no changes to be made to the http auth feature and I fixed the build. Let's roll out 1.3.0-alpha.1

Digenis commented 3 years ago

I'm too busy for a few more days
so I can't review any PRs now.

I'll publish an alpha release with what's already merged. @my8100, ok?

my8100 commented 3 years ago

I think an alpha release is ok. More end-to-end test cases would be added before releasing V1.3

jpmckinney commented 3 years ago

@Digenis has a good summary in #389

1.3 has one last outstanding feature (#325) so that would be an alpha release.

I think we can drop #140 from 1.3 and give the users full control, probably through #197 although I'm not sure we can do this for 1.3, because #187 would need to be fixed too.

If there there are users willing to give feedback for 1.3-alpha we can go on with it. Then I estimate it would take 1-2 weeks to beta and maybe 2-3 weeks to a release candidate.

Regarding 1.4, we'd postpone all its issues for 1.5 and use 1.4 only to bump and drop requirements' versions, because it's been some time since releasing and they are outdated. Even if we target experimental support for py3.10 (twisted is onto it) this would probably take a week or less.

I'm not sure how those issues were selected for inclusion. We might prefer an even more minimal 1.3, to get the already-included features published.

pawelmhm commented 3 years ago

Hi, I'm from Zyte (old Scrapinghub),

My opinion is that we should go with 1.3 this month with the following things:

If there there are users willing to give feedback for 1.3-alpha we can go on with it

I doubt we'll get much feedback and changes won't be that big, so we can just release 1.3 without waiting for user feedback, which may not come. It will delay release without any potential benefit.

Regarding issues in milestone https://github.com/scrapy/scrapyd/milestone/4 clearing this backlog will be hard work, many PR-s are old, many contributors probably forgot about these PR-s, so we will have to rewrite this code again, fix some things, add tests etc. I would leave this backlog for 1.4 or even 2.0. Unless there are volunteers here who can take ownership of these issues and finalize them until end of this month. If yes, please declare who would like to take what.

I see a large problem here with lack of unit tests, and poor test coverage, so I would welcome more pull requests improving test coverage. Also, please, let's stick to the rule of not merging things without unit test coverage.

jpmckinney commented 2 years ago

@pawelmhm Sounds good to me!

mxdev88 commented 2 years ago

Unless there are volunteers here who can take ownership of these issues and finalize them until end of this month. If yes, please declare who would like to take what.

Hi @pawelmhm @jpmckinney

I would like to see #359 in 1.3 if possible. Other people have also expressed interest in the feature. Can it be considered and re-reviewed if necessary ?

jpmckinney commented 2 years ago

@mxdev88 It has now been merged as #418

pawelmhm commented 2 years ago

Released here: https://pypi.org/project/scrapyd/ please report any issues you may find, there was a significant pause between this release and last release (5 years) and several changes, so worth testing everything thoroughly.

Now we should think about 1.3.1 with some minor cleanups that need to be done, there are some deprecation warnings from Scrapy and also six package is no longer needed. Then we can move on to 2.0, for this I would suggest doing new fresh less ugly UI, and some security fixes.