search

conveyal / analysis-backend

Server component of Conveyal Analysis
http://conveyal.com/analysis
MIT License
23 stars 12 forks source link

S3 Seamless Census Sources fail #189

Closed ansoncfit closed 6 years ago

ansoncfit commented 6 years ago

This may have to do with tiles not existing in the ocean. Fetching LODES data for Long Island, for example:

22:35:21.889 [pool-2-thread-3] INFO  c.c.data.census.SeamlessSource - Read 167000 features
22:35:21.907 [pool-2-thread-3] INFO  c.c.data.census.SeamlessSource - Read 168000 features
Exception in thread "pool-2-thread-3" com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 072F0D38F9BCF756; S3 Extended Request ID: IX34emy8ScvS1eGqydWnm8Vby0JxAbLuEbo0o72q2wJnfLSHcl0720IcZcgDwiq9A39nQ6U7CjA=), S3 Extended Request ID: IX34emy8ScvS1eGqydWnm8Vby0JxAbLuEbo0o72q2wJnfLSHcl0720IcZcgDwiq9A39nQ6U7CjA=
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1632)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1058)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4365)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4312)
        at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1415)
        at com.conveyal.taui.grids.SeamlessCensusGridExtractor$S3SeamlessSource.getInputStream(SeamlessCensusGridExtractor.java:40)
        at com.conveyal.data.census.SeamlessSource.extract(SeamlessSource.java:71)
        at com.conveyal.data.census.SeamlessSource.extract(SeamlessSource.java:41)
        at com.conveyal.taui.grids.SeamlessCensusGridExtractor.retrieveAndExtractCensusDataForBounds(SeamlessCensusGridExtractor.java:53)
        at com.conveyal.taui.controllers.OpportunityDatasetController.lambda$downloadLODES$3(OpportunityDatasetController.java:119)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
ansoncfit commented 6 years ago

It appears when we switched to manually triggered downloads, the try/catch block from seamless-census was not ported over to analysis-backend.