This tool scrapes the official COVID-19 Puerto Rico dashboard every hour and keeps tracks of changing metrics in order to help visualize and measure progress.
The new FEED_EXPORT_BATCH_ITEM_COUNT setting allows to deliver output items in batches of up to the specified number of items.
It also serves as a workaround for delayed file delivery, which causes Scrapy to only start item delivery after the crawl has finished when using certain storage backends (S3, FTP, and now GCS).
The base implementation of item loaders has been moved into a separate library, itemloaders, allowing usage from outside Scrapy and a separate release schedule
New ~scrapy.downloadermiddlewares.retry.get_retry_request function to retry requests from spider callbacks
New ~scrapy.signals.headers_received signal that allows stopping downloads early
New Response.protocol <scrapy.http.Response.protocol> attribute
Deprecation removals
Removed all code that was deprecated in 1.7.0 <1.7-deprecations> and had not already been removed in 2.4.0 <2.4-deprecation-removals>. (4901)
Removed support for the SCRAPY_PICKLED_SETTINGS_TO_OVERRIDE environment variable, deprecated in 1.8.0 <1.8-deprecations>. (4912)
Deprecations
The scrapy.utils.py36 module is now deprecated in favor of scrapy.utils.asyncgen. (4900)
New features
Experimental HTTP/2 support <http2> through a new download handler that can be assigned to the https protocol in the DOWNLOAD_HANDLERS setting. (1854, 4769, 5058, 5059, 5066)
The new scrapy.downloadermiddlewares.retry.get_retry_request function may be used from spider callbacks or middlewares to handle the retrying of a request beyond the scenarios that ~scrapy.downloadermiddlewares.retry.RetryMiddleware supports. (3590, 3685, 4902)
The new ~scrapy.signals.headers_received signal gives early access to response headers and allows stopping downloads
<topics-stop-response-download>. (1772, 4897)
The new Response.protocol <scrapy.http.Response.protocol> attribute gives access to the string that identifies the protocol used to download a response. (4878)
Stats <topics-stats> now include the following entries that indicate the number of successes and failures in storing feeds <topics-feed-exports>:
Where <storage type> is the feed storage backend class name, such as ~scrapy.extensions.feedexport.FileFeedStorage or ~scrapy.extensions.feedexport.FTPFeedStorage.
(3947, 4850)
The ~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware spider middleware now logs ignored URLs with INFO logging level
<levels> instead of DEBUG, and it now includes the following entry into stats <topics-stats> to keep track of the number of ignored URLs:
urllength/request_ignored_count
(5036)
The ~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware downloader middleware now logs the number of decompressed responses and the total count of resulting bytes:
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
- `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme
Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
- Update frequency (including time of day and day of week)
- Pull request limits (per update run and/or open at any time)
- Out-of-range updates (receive only lockfile updates, if desired)
- Security updates (receive only security updates, if desired)
Bumps scrapy from 2.2.0 to 2.5.0.
Release notes
Sourced from scrapy's releases.
... (truncated)
Changelog
Sourced from scrapy's changelog.
... (truncated)
Commits
e63188cBump version: 2.4.1 → 2.5.0a71d6ef2.5.0 release notes (#5028)099fb6eMerge pull request #5076 from wRAR/qualname9e3b868Use qualname in middleware handling.8c5a3a5Merge pull request #5073 from elacuesta/fix-urlencode-doseq-arg9e7cbc0Fix type for urlencode's doseq argument127a858Merge pull request #5063 from noviluni/update_usageError_message12f2006Merge pull request #4799 from GeorgeA92/patch-2f0c8d31Merge pull request #4694 from Jgaldos/improve-httpstatus-all-metacc095aaWindows pip installation guide (#4736)Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com): - Update frequency (including time of day and day of week) - Pull request limits (per update run and/or open at any time) - Out-of-range updates (receive only lockfile updates, if desired) - Security updates (receive only security updates, if desired)