Process Accepted, but never actually processes

[tucker-furniss]

I am having a problem running geojobs. The job never initiates. The example from ?geoknife demonstrates the problem:

job <- geoknife(stencil = c(-89,42), fabric = 'prism') check(job)

check(job) $status [1] "Process Accepted" $URL NULL $statusType [1] "ProcessAccepted"

The job stalls out here - no progress even when left overnight.

Session info: geoknife_1.6.4, R version 4.0.3, Windows 10 x64.

[jordansread]

I am sorry but the backend services that support GDP are currently overloaded and the wait is long (and new jobs get added to the queue). You can see status here: https://cida.usgs.gov/gdp/process/dashboard We are asking for some help to manager these jobs as the wait times are too long.

[tucker-furniss]

I see. Thanks for the explanation and the link!

[JodiNorris]

I appreciate this greatly, it could help to move this issue back into the opened category just in defense of the backend server. For example, I've visited this github site more than a dozen times in the last week trying to understand what I could be doing wrong... but because the issue was closed I never saw it in the screen for viewing the issues and it appeared that the most recent one was last year.

Consequence: I've run many geoknife jobs trying to figure out what I'm doing wrong, why half are accepted but don't complete, some of them fail, and cancelling using the cancel command doesn't seem to cancel any of them.

If there is any way I can successfully cancel I'll gladly do that for the jobs on there that are mine.

Best regards, thanks for building a tool that has proven so useful, and thanks for the dashboard link!

Jodi

[jordansread]

I will re-open this and share it w/ the team working on the back-end issues and comment here when there is some better resolution

[JodiNorris]

Any time estimates on when this might be resolved? I'm trying to decide whether to start rewriting scripts using other methods to get updated climate data to a user for her analyses. I know most folks are completely overloaded now, so it's understandable if it will be a while before this is resolved.

Thanks

[jordansread]

@JodiNorris which dataset are you attempting to use? I'm noticing that prismv2 is super backed up in the queue, but others should be ok to get through (the queue is per unique dataset URL, so it doesn't apply to all). Let us know what your specific data task is and we'll see if we can address it sooner.

[JodiNorris]

@jread-usgs Thanks! The dataset that I need the most is ["United States Stage IV Quantitative Precipitation Archive"]

I have alternative methods I can use for Daymet and for the most recent version of PRISM (M3 and D2), since both Daymet and PRISM now have versions that have been corrected for the SNOTEL problem eg https://prism.oregonstate.edu/notices/notice_20191029.php

[dblodgett-usgs]

Hi Jodi -- I'll work with a development team to clean things up tomorrow. Sorry for the trouble.

[dblodgett-usgs]

@JodiNorris -- please try your requests now.

[JodiNorris]

Thanks, I submitted one for a 2-point subset, and I can see 2 processes up there for that time - eg 2021-04-15 09:13

[dblodgett-usgs]

I see your requests and it appears there is another issue. I'm working on it.

[dblodgett-usgs]

OK, I've been investigating two issues. One with PRISM in particular and one more generally with the way geoknife sends in its geometry.

I solved the issue with PRISM but passing geometry in via geoknife rather than using a web service source such as an uploaded or sample polygon set is not working. So if you have geometry you can reference, I THINK this is working. We are still triaging the issue with geoknife's geometry method.

[dblodgett-usgs]

OK I think we are out of the woods. try:

remotes::install_github("usgs-r/geoknife")
library(geoknife)
job <- geoknife(stencil = c(-89,42), fabric = 'prism')
check(job)

[tucker-furniss]

That worked for me! I've tried both PRISM and the MACA dataset and both are working. Thanks very much for fixing the issues!

Note - I initially just reinstalled geoknife from the CRAN repository, and was just getting "process failed". Reinstalled from the github version and it is now working fine.

[jordansread]

Thanks @dblodgett-usgs Thanks @tucker-furniss and @JodiNorris for your patience!

[JodiNorris]

Thanks! It also worked for the ["United States Stage IV Quantitative Precipitation Archive"].

[jordansread]

Fixed in latest CRAN release 🎉