This issue is here to memorialize a fix to issue #405 on labs-community-portal.
The scraper must make two POST requests to LUCATS, each with form data for a given community district, and filtering for either active or completed projects.
The response to the POST is a 302 redirect which triggers a GET to LUCATS with a timestamp as a query parameter, like http://a030-lucats.nyc.gov/lucats/ULURP_Results.aspx?9/19/2017%202:50:59%20PM
Since the scraper is making two calls to LUCATS simultaneously, both requests were being redirected to the same URL, which could only show the results of the latest POST. This explains the inconsistency, as it's possible that the two requests would sometimes fall on the same side of the 1 second boundary, and other times would not.
By offsetting the second request by more than one second, we ensure that the timestamps on the two redirects will be different, and the results for each will be correct.
This issue is here to memorialize a fix to issue #405 on
labs-community-portal.The scraper must make two
POSTrequests to LUCATS, each with form data for a given community district, and filtering for eitheractiveorcompletedprojects.The response to the POST is a 302 redirect which triggers a
GETto LUCATS with a timestamp as a query parameter, likehttp://a030-lucats.nyc.gov/lucats/ULURP_Results.aspx?9/19/2017%202:50:59%20PMSince the scraper is making two calls to LUCATS simultaneously, both requests were being redirected to the same URL, which could only show the results of the latest POST. This explains the inconsistency, as it's possible that the two requests would sometimes fall on the same side of the 1 second boundary, and other times would not.
By offsetting the second request by more than one second, we ensure that the timestamps on the two redirects will be different, and the results for each will be correct.