search

ClasicRando / web_rest_scraper

Single page static web application to scrape data from an ArcGIS REST service
https://ClasicRando.github.io/web_rest_scraper/
MIT License
1 stars 0 forks source link

UST Table for REST Scrape, unsuccessful #20

Closed ERISjh closed 1 year ago

ERISjh commented 1 year ago

Good afternoon! I would like to please ask about a REST scrape of the following Service Base URL: https://services.arcgis.com/cJ9YHowT8TU7DUyn/ArcGIS/rest/services/UST_Finder_Feature_Layer_2/FeatureServer/4 How can I return CSV Rest Scrape result with full/complete count for the UST Table (not limited to only 2000 records)? Or because this is as a table and not a true layer, will a REST Scrape not work properly? Thank you for your time in advance.

ClasicRando commented 1 year ago

Hi!

The issue here appears to be that the service supports query cache hints which makes my standard count check always return 0 if the "cacheHint=false" option is not provided in the request. I should be able to have a fix created sometime later tomorrow.

ERISjh commented 1 year ago

Good Morning, Steven!

I hope you are doing very well. Thanks so much for your review, response and further time to help with my submitted question. I really appreciate it!

Jennife​r Horan

Data Collection Manager | ERIS

P: 416.510.5204 Ext. 93121 C: 512.289.9410 | W: erisinfo.com Tw: @ERISInformationhttps://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FERISInformation&data=05%7C01%7Cjhoran%40erisinfo.com%7C8a334621713a404fbef508da4b2f8c88%7C5a6f30998a9543e9a941e2c7022c9f11%7C0%7C0%7C637904964860347979%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iW%2FhAeHxheC8I196FnWa4q6yEFaV%2Fvsg3nIHs4ybxsQ%3D&reserved=0 | Li: @ERIShttps://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fenvironmental-risk-information-services%2F%3FviewAsMember%3Dtrue&data=05%7C01%7Cjhoran%40erisinfo.com%7C8a334621713a404fbef508da4b2f8c88%7C5a6f30998a9543e9a941e2c7022c9f11%7C0%7C0%7C637904964860347979%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=g0%2BFE5So6TPeyasrUrtgUln4HT5vKMgpYBMdjmp2HEI%3D&reserved=0 https://www.erisinfo.com/ca/[cid:b6a9ae07-86f4-4c39-8122-22a98b263aaf]https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgo.erisinfo.com%2Fl%2F506521%2F2022-05-09%2Fcn9dh2&data=05%7C01%7Cjhoran%40erisinfo.com%7C8a334621713a404fbef508da4b2f8c88%7C5a6f30998a9543e9a941e2c7022c9f11%7C0%7C0%7C637904964860347979%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iLn4OJaxmJ%2FlSATej2WcWf1z9dwWz5sZVbKKlWTAMT8%3D&reserved=0

From: Steven Thomson @.> Sent: Monday, February 13, 2023 5:00 PM To: ClasicRando/web_rest_scraper @.> Cc: Jennifer Horan @.>; Author @.> Subject: Re: [ClasicRando/web_rest_scraper] UST Table for REST Scrape, unsuccessful (Issue #20)

Hi!

The issue here appears to be that the service supports query cache hints which makes my standard count check always return 0 if the "cacheHint=false" option is not provided in the request. I should be able to have a fix created sometime later tomorrow.

— Reply to this email directly, view it on GitHubhttps://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FClasicRando%2Fweb_rest_scraper%2Fissues%2F20%23issuecomment-1428821353&data=05%7C01%7Cjhoran%40erisinfo.com%7C462a80f3607745c248d708db0e161e55%7C5a6f30998a9543e9a941e2c7022c9f11%7C0%7C0%7C638119260379502686%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=b5OT%2FxbndXxp4D1mQWsNHWVaVyywWwzie3D7E8FDwfQ%3D&reserved=0, or unsubscribehttps://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FA54AKIKAXYDUK3QSBWX3GA3WXK4JHANCNFSM6AAAAAAU2VLVRE&data=05%7C01%7Cjhoran%40erisinfo.com%7C462a80f3607745c248d708db0e161e55%7C5a6f30998a9543e9a941e2c7022c9f11%7C0%7C0%7C638119260379502686%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qxcgaB2vAMTEaMcvLb5xQe6E9iFCrz%2FhxeIvBk%2BPaEU%3D&reserved=0. You are receiving this because you authored the thread.Message ID: @.***>

ClasicRando commented 1 year ago

I have pushed the change of #21 to resolve this issue. You can now scrape this service but it's quite large so give it some time to complete all the requests and let me know if anything else comes up!

ERISjh commented 1 year ago

Good Morning Steven!

Thanks so much for your time to help me with this issue. I have the scrape in process, its been going for a while already - but like you said, its over 2 mil for the feature count so I'll be patient.

Jennife​r Horan

Data Collection Manager | ERIS

P: 416.510.5204 Ext. 93121 C: 512.289.9410 | W: erisinfo.com Tw: @ERISInformationhttps://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FERISInformation&data=05%7C01%7Cjhoran%40erisinfo.com%7C8a334621713a404fbef508da4b2f8c88%7C5a6f30998a9543e9a941e2c7022c9f11%7C0%7C0%7C637904964860347979%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iW%2FhAeHxheC8I196FnWa4q6yEFaV%2Fvsg3nIHs4ybxsQ%3D&reserved=0 | Li: @ERIShttps://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fenvironmental-risk-information-services%2F%3FviewAsMember%3Dtrue&data=05%7C01%7Cjhoran%40erisinfo.com%7C8a334621713a404fbef508da4b2f8c88%7C5a6f30998a9543e9a941e2c7022c9f11%7C0%7C0%7C637904964860347979%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=g0%2BFE5So6TPeyasrUrtgUln4HT5vKMgpYBMdjmp2HEI%3D&reserved=0 https://www.erisinfo.com/ca/[cid:e76bc7f5-0dad-4773-843b-3b915502d046]https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgo.erisinfo.com%2Fl%2F506521%2F2022-05-09%2Fcn9dh2&data=05%7C01%7Cjhoran%40erisinfo.com%7C8a334621713a404fbef508da4b2f8c88%7C5a6f30998a9543e9a941e2c7022c9f11%7C0%7C0%7C637904964860347979%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iLn4OJaxmJ%2FlSATej2WcWf1z9dwWz5sZVbKKlWTAMT8%3D&reserved=0

From: Steven Thomson @.> Sent: Tuesday, February 14, 2023 8:39 PM To: ClasicRando/web_rest_scraper @.> Cc: Jennifer Horan @.>; Author @.> Subject: Re: [ClasicRando/web_rest_scraper] UST Table for REST Scrape, unsuccessful (Issue #20)

Closed #20https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FClasicRando%2Fweb_rest_scraper%2Fissues%2F20&data=05%7C01%7Cjhoran%40erisinfo.com%7C7057996c3f494434c83d08db0efde30e%7C5a6f30998a9543e9a941e2c7022c9f11%7C0%7C0%7C638120255823576999%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nVsmeepTGwa7g0pwjNIBIHj2ldfPTCNL6JRi4bwB89g%3D&reserved=0 as completed.

— Reply to this email directly, view it on GitHubhttps://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FClasicRando%2Fweb_rest_scraper%2Fissues%2F20%23event-8521523266&data=05%7C01%7Cjhoran%40erisinfo.com%7C7057996c3f494434c83d08db0efde30e%7C5a6f30998a9543e9a941e2c7022c9f11%7C0%7C0%7C638120255823576999%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nHM3aZOSEH0SEaVxuKWmFlsp7G7Q52DWMRs0%2BFnsnuk%3D&reserved=0, or unsubscribehttps://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FA54AKIK53B6GFB7LREL7F5DWXQ6WXANCNFSM6AAAAAAU2VLVRE&data=05%7C01%7Cjhoran%40erisinfo.com%7C7057996c3f494434c83d08db0efde30e%7C5a6f30998a9543e9a941e2c7022c9f11%7C0%7C0%7C638120255823733212%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=GpcIa5o19LIYWq3nh0hIaFZQAGcds59T%2Fj3hjah5lb8%3D&reserved=0. You are receiving this because you authored the thread.Message ID: @.***>