search

NEONScience / NEON-utilities

Utilities and scripts for working with NEON data. Currently: an R package with functions to join (stack) the month-by-site files in downloaded NEON data, to convert data to geoCSV format, and to download data from the API.
GNU Affero General Public License v3.0
57 stars 36 forks source link

The API does not recognize 'include.provisional' #133

Closed JohnStranzl closed 1 month ago

JohnStranzl commented 1 month ago

Function zipbyproducts

Describe the bug This is the error returned by neonUtilities when I use the 'provisional' data parameter: R[write to console]: Error in !include.provisional : invalid argument type

If I DON'T use the provisional data parameter, this is the error I get: Finding available files |======================================================================| 100% R[write to console]: Error in getZipUrls(month.urls, avg = avg, package = package, dpID = dpID, : All files found were provisional. Modify download query (dates and/or sites) or, if you want to use provisional data, use input parameter include.provisional=TRUE. R[write to console]: In addition: R[write to console]: Warning messages: R[write to console]: 1: R[write to console]: In strptime(x, fmt, tz = "GMT") : R[write to console]: unknown timezone 'America/New_York' R[write to console]: 2: R[write to console]: In strptime(x, fmt, tz = "GMT") : R[write to console]: unknown timezone 'GMT' R[write to console]: 3: R[write to console]: In strptime(x, fmt, tz = "GMT") : R[write to console]: unknown timezone 'America/New_York'

To Reproduce utils.capture_output(neonUtilities.zipsByProduct(dpID = strProduct, \ site = SiteCode, \ savepath = downloadsFilePath, \ startdate = strStartDate, \ enddate = strEndDate, \ package = 'basic', \ include_provisional = 'TRUE', \ check_size = 'FALSE'))

Expected behavior This function worked fine for a couple of years until much of the data started being label as provisional. For most of 2024, this function is not working.

System (please complete the following information):

Additional context The package loads correctly: neonUtilities = rpackages.importr('neonUtilities') utils = importr('utils')

I am using the specified server: SERVER = 'http://data.neonscience.org/api/v0/'

cklunch commented 1 month ago

@JohnStranzl TRUE and FALSE shouldn't be quoted, those are pre-defined terms in R. Try:

utils.capture_output(neonUtilities.zipsByProduct(dpID = strProduct,
site = SiteCode,
savepath = downloadsFilePath,
startdate = strStartDate,
enddate = strEndDate,
package = 'basic',
include_provisional = TRUE,
check_size = FALSE))

Let us know if that resolves the error. Thank you!

JohnStranzl commented 1 month ago

If I don't put TRUE or FALSE in quotes in my python script, the API doesn't attempt to download it at all. If I at least put in in quotes, it attempts to start the download process but says it doesn't recognize 'include_provisional'. This whole issue seems to have arisen when more and more data was being marked as provisional.

[image: image.png]

On Mon, Jul 29, 2024 at 9:06 AM Claire Lunch @.***> wrote:

@JohnStranzl https://github.com/JohnStranzl TRUE and FALSE shouldn't be quoted, those are pre-defined terms in R. Try:

utils.capture_output(neonUtilities.zipsByProduct(dpID = strProduct, site = SiteCode, savepath = downloadsFilePath, startdate = strStartDate, enddate = strEndDate, package = 'basic', include_provisional = TRUE, check_size = FALSE))

Let us know if that resolves the error. Thank you!

— Reply to this email directly, view it on GitHub https://github.com/NEONScience/NEON-utilities/issues/133#issuecomment-2255887458, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIRROXTHLNYA5MVAWES7RDZOY44RAVCNFSM6AAAAABLTJR4JKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJVHA4DONBVHA . You are receiving this because you were mentioned.Message ID: @.***>

JohnStranzl commented 1 month ago

Here is the function. As soon as I execute the 'zipByProducts', it immediately throws an exception:

def FetchData(self, SiteCode, strProduct, strStartDate, strEndDate, downloadsFilePath):

nError = 0

if os.path.exists(downloadsFilePath) == False:
    os.makedirs(downloadsFilePath)

foldername = strProduct.split('.')[1].zfill(5) + ' -' +

strProduct.split('.')[2].split(':')[1] strProduct = 'DP1.' + strProduct.split('.')[1] + '.001'

progressBar = QProgressWheel()
progressBar.setRange(0, 100)
progressBar.show()

progressBar.setValue(20)

neonUtilities = rpackages.importr('neonUtilities')
utils = importr('utils')

try:
    utils.capture_output(neonUtilities.zipsByProduct(dpID = strProduct, \
                                                     site = SiteCode, \
                                                     savepath =

downloadsFilePath, \ startdate = strStartDate, \ enddate = strEndDate, \ package = 'basic', \

include_provisional = TRUE, \ check_size = FALSE))

    progressBar.setValue(40)

    # PATH WHERE THE zipsByProduct PLACED THE DOWNLOADED ZIP FILES
    myFolderPath = downloadsFilePath + '\\filesToStack' +

strProduct.split('.')[1].zfill(5)

    # PATH WHERE WE WANT TO PLACE THE STACKED FILES (i.e.,

CONCATENATED MONTHLY DATA FILES) WILL BE STORED mySavePath = downloadsFilePath + '\' + foldername if os.path.exists(mySavePath): shutil.rmtree(mySavePath)

    # USE AS MANY CORES ARE AVAILABLE FOR STACKING THE DATA (i.e.,

UNZIP ALL THE INDIVIDUAL MONTHS DOWNLOADED ZIP

FILES AND CONCATENATE INTO ONE CSV FILE)

    nMyCores = multiprocessing.cpu_count()

    progressBar.setValue(60)

    neonUtilities.stackByTable(filepath=myFolderPath,

savepath=mySavePath, nCores=nMyCores)

    # IF ALL ZIPPED FILES WERE STACKED PROPERLY, AND THE FUNCTION

stackByTable REMOVES THE ZIP FILES ONCE ALL

FILES ARE CONCATENATED, WE CAN DELETE WHAT NOW SHOULD BE THE

EMPTY ZIP FILE DOWNLOAD FOLDER. if os.path.exists(myFolderPath): shutil.rmtree(myFolderPath)

    # LET'S REMOVE ONE LEVEL OF INDIRECTION AND MOVE THE FILES

STACKED BY stackByTable TO THE ROOT PRODUCT FOLDER IN THE

DOWNLOAD DIRECTORY

    src = downloadsFilePath + '\\' + foldername + '\\stackedFiles'
    self.dest = downloadsFilePath + '\\' + foldername
    if os.path.exists(src) and os.path.exists(self.dest):
        filenames = os.listdir(src)
        for filename in filenames:
            shutil.move(os.path.join(src, filename), self.dest)

        # NOW WE REMOVE THE EMPTY STACK FOLDER SINCE THE FILES

HAVE BEEN MOVED shutil.rmtree(src)

    progressBar.setValue(100)

except:
    nError = -1

progressBar.close()
del progressBar

return nError

On Mon, Jul 29, 2024 at 1:11 PM John Stranzl @.***> wrote:

If I don't put TRUE or FALSE in quotes in my python script, the API doesn't attempt to download it at all. If I at least put in in quotes, it attempts to start the download process but says it doesn't recognize 'include_provisional'. This whole issue seems to have arisen when more and more data was being marked as provisional.

[image: image.png]

On Mon, Jul 29, 2024 at 9:06 AM Claire Lunch @.***> wrote:

@JohnStranzl https://github.com/JohnStranzl TRUE and FALSE shouldn't be quoted, those are pre-defined terms in R. Try:

utils.capture_output(neonUtilities.zipsByProduct(dpID = strProduct, site = SiteCode, savepath = downloadsFilePath, startdate = strStartDate, enddate = strEndDate, package = 'basic', include_provisional = TRUE, check_size = FALSE))

Let us know if that resolves the error. Thank you!

— Reply to this email directly, view it on GitHub https://github.com/NEONScience/NEON-utilities/issues/133#issuecomment-2255887458, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIRROXTHLNYA5MVAWES7RDZOY44RAVCNFSM6AAAAABLTJR4JKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJVHA4DONBVHA . You are receiving this because you were mentioned.Message ID: @.***>

JohnStranzl commented 1 month ago

I was able to trap on the call stack and get the following error. It seems to be related to GMT.

Backend QtAgg is interactive backend. Turning interactive mode on. function (tab, useFasttime = FALSE) { tBgnErr <- FALSE tEndErr <- FALSE if (useFasttime) { tabBP <- try(fasttime::fastPOSIXct(tab$timeBgn, tz = "GMT"), silent = T) tabEP <- try(fasttime::fastPOSIXct(tab$timeEnd, tz = "GMT"), silent = T) } else { tabBP <- try(as.POSIXct(tab$timeBgn, format = "%Y-%m-%dT%H:%M:%OS", tz = "GMT"), silent = T) tabEP <- try(as.POSIXct(tab$timeEnd, format = "%Y-%m-%dT%H:%M:%OS", tz = "GMT"), silent = T) } if (any(c(inherits(tabBP, "try-error"), all(is.na(tabBP))))) { tBgnErr <- TRUE } if (any(c(inherits(tabBP, "try-error"), all(is.na(tabEP))))) { tEndErr <- TRUE } tabN <- tab err <- FALSE if (tBgnErr) { err <- TRUE } else { tabN$timeBgn <- tabBP } if (tEndErr) { err <- TRUE } else { tabN$timeEnd <- tabEP } if (err) { tabN <- tabN } else { dayDiff <- base::as.difftime(tabEP - tabBP) secDiff <- base::abs(base::as.numeric(dayDiff, units = "secs")) dayDup <- which(secDiff >= 86399) if (length(dayDup) == 0) { tabN <- tabN } else { tabN <- tabN[-dayDup, ] } } return(list(tabN, err)) } <bytecode: 0x000001d96e7f4e88>

On Mon, Jul 29, 2024 at 1:15 PM John Stranzl ***@***.***> wrote: > Here is the function. As soon as I execute the 'zipByProducts', it > immediately throws an exception: > > def FetchData(self, SiteCode, strProduct, strStartDate, strEndDate, downloadsFilePath): > > nError = 0 > > if os.path.exists(downloadsFilePath) == False: > os.makedirs(downloadsFilePath) > > foldername = strProduct.split('.')[1].zfill(5) + ' -' + strProduct.split('.')[2].split(':')[1] > strProduct = 'DP1.' + strProduct.split('.')[1] + '.001' > > progressBar = QProgressWheel() > progressBar.setRange(0, 100) > progressBar.show() > > progressBar.setValue(20) > > neonUtilities = rpackages.importr('neonUtilities') > utils = importr('utils') > > try: > utils.capture_output(neonUtilities.zipsByProduct(dpID = strProduct, \ > site = SiteCode, \ > savepath = downloadsFilePath, \ > startdate = strStartDate, \ > enddate = strEndDate, \ > package = 'basic', \ > include_provisional = TRUE, \ > check_size = FALSE)) > > progressBar.setValue(40) > > # PATH WHERE THE zipsByProduct PLACED THE DOWNLOADED ZIP FILES > myFolderPath = downloadsFilePath + '\\filesToStack' + strProduct.split('.')[1].zfill(5) > > # PATH WHERE WE WANT TO PLACE THE STACKED FILES (i.e., CONCATENATED MONTHLY DATA FILES) WILL BE STORED > mySavePath = downloadsFilePath + '\\' + foldername > if os.path.exists(mySavePath): > shutil.rmtree(mySavePath) > > # USE AS MANY CORES ARE AVAILABLE FOR STACKING THE DATA (i.e., UNZIP ALL THE INDIVIDUAL MONTHS DOWNLOADED ZIP > # FILES AND CONCATENATE INTO ONE CSV FILE) > nMyCores = multiprocessing.cpu_count() > > progressBar.setValue(60) > > neonUtilities.stackByTable(filepath=myFolderPath, savepath=mySavePath, nCores=nMyCores) > > # IF ALL ZIPPED FILES WERE STACKED PROPERLY, AND THE FUNCTION stackByTable REMOVES THE ZIP FILES ONCE ALL > # FILES ARE CONCATENATED, WE CAN DELETE WHAT NOW SHOULD BE THE EMPTY ZIP FILE DOWNLOAD FOLDER. > if os.path.exists(myFolderPath): > shutil.rmtree(myFolderPath) > > # LET'S REMOVE ONE LEVEL OF INDIRECTION AND MOVE THE FILES STACKED BY stackByTable TO THE ROOT PRODUCT FOLDER IN THE > # DOWNLOAD DIRECTORY > src = downloadsFilePath + '\\' + foldername + '\\stackedFiles' > self.dest = downloadsFilePath + '\\' + foldername > if os.path.exists(src) and os.path.exists(self.dest): > filenames = os.listdir(src) > for filename in filenames: > shutil.move(os.path.join(src, filename), self.dest) > > # NOW WE REMOVE THE EMPTY STACK FOLDER SINCE THE FILES HAVE BEEN MOVED > shutil.rmtree(src) > > progressBar.setValue(100) > > except: > nError = -1 > > progressBar.close() > del progressBar > > return nError > > > On Mon, Jul 29, 2024 at 1:11 PM John Stranzl ***@***.***> > wrote: > >> If I don't put TRUE or FALSE in quotes in my python script, the API >> doesn't attempt to download it at all. If I at least put in in quotes, it >> attempts to start the download process but says it doesn't recognize >> 'include_provisional'. This whole issue seems to have arisen when more and >> more data was being marked as provisional. >> >> [image: image.png] >> >> >> >> On Mon, Jul 29, 2024 at 9:06 AM Claire Lunch ***@***.***> >> wrote: >> >>> @JohnStranzl >>> TRUE and FALSE shouldn't be quoted, those are pre-defined terms in R. >>> Try: >>> >>> utils.capture_output(neonUtilities.zipsByProduct(dpID = strProduct, >>> site = SiteCode, >>> savepath = downloadsFilePath, >>> startdate = strStartDate, >>> enddate = strEndDate, >>> package = 'basic', >>> include_provisional = TRUE, >>> check_size = FALSE)) >>> >>> Let us know if that resolves the error. Thank you! >>> >>> — >>> Reply to this email directly, view it on GitHub >>> , >>> or unsubscribe >>> >>> . >>> You are receiving this because you were mentioned.Message ID: >>> ***@***.***> >>> >>
JohnStranzl commented 1 month ago

Here are the start and end dates fed to zipByProducts: [image: image.png]

On Mon, Jul 29, 2024 at 1:22 PM John Stranzl @.***> wrote:

I was able to trap on the call stack and get the following error. It seems to be related to GMT.

Backend QtAgg is interactive backend. Turning interactive mode on. function (tab, useFasttime = FALSE) { tBgnErr <- FALSE tEndErr <- FALSE if (useFasttime) { tabBP <- try(fasttime::fastPOSIXct(tab$timeBgn, tz = "GMT"), silent = T) tabEP <- try(fasttime::fastPOSIXct(tab$timeEnd, tz = "GMT"), silent = T) } else { tabBP <- try(as.POSIXct(tab$timeBgn, format = "%Y-%m-%dT%H:%M:%OS", tz = "GMT"), silent = T) tabEP <- try(as.POSIXct(tab$timeEnd, format = "%Y-%m-%dT%H:%M:%OS", tz = "GMT"), silent = T) } if (any(c(inherits(tabBP, "try-error"), all(is.na(tabBP))))) { tBgnErr <- TRUE } if (any(c(inherits(tabBP, "try-error"), all(is.na(tabEP))))) { tEndErr <- TRUE } tabN <- tab err <- FALSE if (tBgnErr) { err <- TRUE } else { tabN$timeBgn <- tabBP } if (tEndErr) { err <- TRUE } else { tabN$timeEnd <- tabEP } if (err) { tabN <- tabN } else { dayDiff <- base::as.difftime(tabEP - tabBP) secDiff <- base::abs(base::as.numeric(dayDiff, units = "secs")) dayDup <- which(secDiff >= 86399) if (length(dayDup) == 0) { tabN <- tabN } else { tabN <- tabN[-dayDup, ] } } return(list(tabN, err)) } <bytecode: 0x000001d96e7f4e88>

On Mon, Jul 29, 2024 at 1:15 PM John Stranzl ***@***.***> wrote: > Here is the function. As soon as I execute the 'zipByProducts', it > immediately throws an exception: > > def FetchData(self, SiteCode, strProduct, strStartDate, strEndDate, downloadsFilePath): > > nError = 0 > > if os.path.exists(downloadsFilePath) == False: > os.makedirs(downloadsFilePath) > > foldername = strProduct.split('.')[1].zfill(5) + ' -' + strProduct.split('.')[2].split(':')[1] > strProduct = 'DP1.' + strProduct.split('.')[1] + '.001' > > progressBar = QProgressWheel() > progressBar.setRange(0, 100) > progressBar.show() > > progressBar.setValue(20) > > neonUtilities = rpackages.importr('neonUtilities') > utils = importr('utils') > > try: > utils.capture_output(neonUtilities.zipsByProduct(dpID = strProduct, \ > site = SiteCode, \ > savepath = downloadsFilePath, \ > startdate = strStartDate, \ > enddate = strEndDate, \ > package = 'basic', \ > include_provisional = TRUE, \ > check_size = FALSE)) > > progressBar.setValue(40) > > # PATH WHERE THE zipsByProduct PLACED THE DOWNLOADED ZIP FILES > myFolderPath = downloadsFilePath + '\\filesToStack' + strProduct.split('.')[1].zfill(5) > > # PATH WHERE WE WANT TO PLACE THE STACKED FILES (i.e., CONCATENATED MONTHLY DATA FILES) WILL BE STORED > mySavePath = downloadsFilePath + '\\' + foldername > if os.path.exists(mySavePath): > shutil.rmtree(mySavePath) > > # USE AS MANY CORES ARE AVAILABLE FOR STACKING THE DATA (i.e., UNZIP ALL THE INDIVIDUAL MONTHS DOWNLOADED ZIP > # FILES AND CONCATENATE INTO ONE CSV FILE) > nMyCores = multiprocessing.cpu_count() > > progressBar.setValue(60) > > neonUtilities.stackByTable(filepath=myFolderPath, savepath=mySavePath, nCores=nMyCores) > > # IF ALL ZIPPED FILES WERE STACKED PROPERLY, AND THE FUNCTION stackByTable REMOVES THE ZIP FILES ONCE ALL > # FILES ARE CONCATENATED, WE CAN DELETE WHAT NOW SHOULD BE THE EMPTY ZIP FILE DOWNLOAD FOLDER. > if os.path.exists(myFolderPath): > shutil.rmtree(myFolderPath) > > # LET'S REMOVE ONE LEVEL OF INDIRECTION AND MOVE THE FILES STACKED BY stackByTable TO THE ROOT PRODUCT FOLDER IN THE > # DOWNLOAD DIRECTORY > src = downloadsFilePath + '\\' + foldername + '\\stackedFiles' > self.dest = downloadsFilePath + '\\' + foldername > if os.path.exists(src) and os.path.exists(self.dest): > filenames = os.listdir(src) > for filename in filenames: > shutil.move(os.path.join(src, filename), self.dest) > > # NOW WE REMOVE THE EMPTY STACK FOLDER SINCE THE FILES HAVE BEEN MOVED > shutil.rmtree(src) > > progressBar.setValue(100) > > except: > nError = -1 > > progressBar.close() > del progressBar > > return nError > > > On Mon, Jul 29, 2024 at 1:11 PM John Stranzl ***@***.***> > wrote: > >> If I don't put TRUE or FALSE in quotes in my python script, the API >> doesn't attempt to download it at all. If I at least put in in quotes, it >> attempts to start the download process but says it doesn't recognize >> 'include_provisional'. This whole issue seems to have arisen when more and >> more data was being marked as provisional. >> >> [image: image.png] >> >> >> >> On Mon, Jul 29, 2024 at 9:06 AM Claire Lunch ***@***.***> >> wrote: >> >>> @JohnStranzl >>> TRUE and FALSE shouldn't be quoted, those are pre-defined terms in R. >>> Try: >>> >>> utils.capture_output(neonUtilities.zipsByProduct(dpID = strProduct, >>> site = SiteCode, >>> savepath = downloadsFilePath, >>> startdate = strStartDate, >>> enddate = strEndDate, >>> package = 'basic', >>> include_provisional = TRUE, >>> check_size = FALSE)) >>> >>> Let us know if that resolves the error. Thank you! >>> >>> — >>> Reply to this email directly, view it on GitHub >>> , >>> or unsubscribe >>> >>> . >>> You are receiving this because you were mentioned.Message ID: >>> ***@***.***> >>> >>
cklunch commented 1 month ago

@JohnStranzl Thanks for the additional context, I was forgetting the issues around transferring boolean values between R and Python with rpy2. Unfortunately quotes work in some situations and not others, it depends on the details of the code on the R end. This runs successfully for me:

neonUtilities.zipsByProduct(dpID = strProduct,
site = SiteCode,
savepath = downloadsFilePath,
startdate = strStartDate,
enddate = strEndDate,
package = 'basic',
include_provisional = robjects.vectors.BoolVector([True]),
check_size = 'FALSE')

We're currently working toward a Python version of neonUtilities, aiming for release to the public this fall - once that is available, you'll be able to code directly in Python and avoid rpy2 entirely! Keep an eye on Data Notifications for the announcement, or you can sign up for Data Notification emails with a NEON user account.

Some background info on opting in for Provisional data here.

JohnStranzl commented 1 month ago

Well... It no longer crashes but it doesn't download any data. I will see if I can get you any additional information. Here are the code changes I made based upon your previous email. I also had to change 'check_size' as shown in the following code.

utils.capture_output(neonUtilities.zipsByProduct(dpID = strProduct, \ site = SiteCode, \ savepath = downloadsFilePath, \ startdate = strStartDate, \ enddate = strEndDate, \ package = 'basic', \ include_provisional = robjects.vectors.BoolVector([True]), \ check_size = robjects.vectors.BoolVector([False])))

On Mon, Jul 29, 2024 at 2:03 PM Claire Lunch @.***> wrote:

@JohnStranzl https://github.com/JohnStranzl Thanks for the additional context, I was forgetting the issues around transferring boolean values between R and Python with rpy2. Unfortunately quotes work in some situations and not others, it depends on the details of the code on the R end. This runs successfully for me:

neonUtilities.zipsByProduct(dpID = strProduct, site = SiteCode, savepath = downloadsFilePath, startdate = strStartDate, enddate = strEndDate, package = 'basic', include_provisional = robjects.vectors.BoolVector([True]), check_size = 'FALSE')

We're currently working toward a Python version of neonUtilities, aiming for release to the public this fall - once that is available, you'll be able to code directly in Python and avoid rpy2 entirely! Keep an eye on Data Notifications https://www.neonscience.org/data-samples/data-notifications for the announcement, or you can sign up for Data Notification emails with a NEON user account https://www.neonscience.org/about/user-accounts.

Some background info on opting in for Provisional data here https://www.neonscience.org/impact/observatory-blog/change-default-neon-data-download-behavior .

— Reply to this email directly, view it on GitHub https://github.com/NEONScience/NEON-utilities/issues/133#issuecomment-2256583673, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIRROUV2QQTT24BEOEXNQDZOZ7XJAVCNFSM6AAAAABLTJR4JKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJWGU4DGNRXGM . You are receiving this because you were mentioned.Message ID: @.***>

JohnStranzl commented 1 month ago

zipByProducts didn't download anything even though it didn't crash. I asked for Nitrate data from March 2024 through July 2024 and it did not download anything. I changed the range from March 2024 to June 2024 (because there's no data available for July) and it still did not download anything.

In the past few months, what has changed? This code was working find for me for 2+ years? Everything seemed to go wrong when NEON started marking more and more data provisional.

I am using this for my dissertation work. So you can imagine how this is putting a cramp in my work. :(

-John

On Mon, Jul 29, 2024 at 6:50 PM John Stranzl @.***> wrote:

Well... It no longer crashes but it doesn't download any data. I will see if I can get you any additional information. Here are the code changes I made based upon your previous email. I also had to change 'check_size' as shown in the following code.

utils.capture_output(neonUtilities.zipsByProduct(dpID = strProduct, \ site = SiteCode, \ savepath = downloadsFilePath, \ startdate = strStartDate, \ enddate = strEndDate, \ package = 'basic', \ include_provisional = robjects.vectors.BoolVector([True]), \ check_size = robjects.vectors.BoolVector([False])))

On Mon, Jul 29, 2024 at 2:03 PM Claire Lunch @.***> wrote:

@JohnStranzl https://github.com/JohnStranzl Thanks for the additional context, I was forgetting the issues around transferring boolean values between R and Python with rpy2. Unfortunately quotes work in some situations and not others, it depends on the details of the code on the R end. This runs successfully for me:

neonUtilities.zipsByProduct(dpID = strProduct, site = SiteCode, savepath = downloadsFilePath, startdate = strStartDate, enddate = strEndDate, package = 'basic', include_provisional = robjects.vectors.BoolVector([True]), check_size = 'FALSE')

We're currently working toward a Python version of neonUtilities, aiming for release to the public this fall - once that is available, you'll be able to code directly in Python and avoid rpy2 entirely! Keep an eye on Data Notifications https://www.neonscience.org/data-samples/data-notifications for the announcement, or you can sign up for Data Notification emails with a NEON user account https://www.neonscience.org/about/user-accounts.

Some background info on opting in for Provisional data here https://www.neonscience.org/impact/observatory-blog/change-default-neon-data-download-behavior .

— Reply to this email directly, view it on GitHub https://github.com/NEONScience/NEON-utilities/issues/133#issuecomment-2256583673, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIRROUV2QQTT24BEOEXNQDZOZ7XJAVCNFSM6AAAAABLTJR4JKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJWGU4DGNRXGM . You are receiving this because you were mentioned.Message ID: @.***>

JohnStranzl commented 1 month ago

So I put the quotes around the True and False, and at the command prompt it asks if I want to download the data. So progress of a sort? But a) I don't want to be prompted and b) even though I said yes at the prompt, it did not download anything, still.

utils.capture_output(neonUtilities.zipsByProduct(dpID = strProduct, \ site = SiteCode, \ savepath = downloadsFilePath, \ startdate = strStartDate, \ enddate = strEndDate, \ package = 'basic', \ include_provisional = robjects.vectors.BoolVector(['True']), \ check_size = robjects.vectors.BoolVector(['False'])))

On Mon, Jul 29, 2024 at 7:00 PM John Stranzl @.***> wrote:

zipByProducts didn't download anything even though it didn't crash. I asked for Nitrate data from March 2024 through July 2024 and it did not download anything. I changed the range from March 2024 to June 2024 (because there's no data available for July) and it still did not download anything.

In the past few months, what has changed? This code was working find for me for 2+ years? Everything seemed to go wrong when NEON started marking more and more data provisional.

I am using this for my dissertation work. So you can imagine how this is putting a cramp in my work. :(

-John

On Mon, Jul 29, 2024 at 6:50 PM John Stranzl @.***> wrote:

Well... It no longer crashes but it doesn't download any data. I will see if I can get you any additional information. Here are the code changes I made based upon your previous email. I also had to change 'check_size' as shown in the following code.

utils.capture_output(neonUtilities.zipsByProduct(dpID = strProduct, \ site = SiteCode, \ savepath = downloadsFilePath, \ startdate = strStartDate, \ enddate = strEndDate, \ package = 'basic', \ include_provisional = robjects.vectors.BoolVector([True]), \ check_size = robjects.vectors.BoolVector([False])))

On Mon, Jul 29, 2024 at 2:03 PM Claire Lunch @.***> wrote:

@JohnStranzl https://github.com/JohnStranzl Thanks for the additional context, I was forgetting the issues around transferring boolean values between R and Python with rpy2. Unfortunately quotes work in some situations and not others, it depends on the details of the code on the R end. This runs successfully for me:

neonUtilities.zipsByProduct(dpID = strProduct, site = SiteCode, savepath = downloadsFilePath, startdate = strStartDate, enddate = strEndDate, package = 'basic', include_provisional = robjects.vectors.BoolVector([True]), check_size = 'FALSE')

We're currently working toward a Python version of neonUtilities, aiming for release to the public this fall - once that is available, you'll be able to code directly in Python and avoid rpy2 entirely! Keep an eye on Data Notifications https://www.neonscience.org/data-samples/data-notifications for the announcement, or you can sign up for Data Notification emails with a NEON user account https://www.neonscience.org/about/user-accounts.

Some background info on opting in for Provisional data here https://www.neonscience.org/impact/observatory-blog/change-default-neon-data-download-behavior .

— Reply to this email directly, view it on GitHub https://github.com/NEONScience/NEON-utilities/issues/133#issuecomment-2256583673, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIRROUV2QQTT24BEOEXNQDZOZ7XJAVCNFSM6AAAAABLTJR4JKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJWGU4DGNRXGM . You are receiving this because you were mentioned.Message ID: @.***>

JohnStranzl commented 1 month ago

One minute it says it downloaded and the next it is saying it didn't. And how do I get rid of the prompt?

Output from API Continuing will download files totaling approximately 0.208967 MB. Do you want to proceed y/n: >? y Downloading 1 files | | 0% NEON.D03.BARC.DP1.20033.001.2024-04.basic.20240520T151453Z.PROVISIONAL.zip could not be downloaded. Re-attempting. NEON.D03.BARC.DP1.20033.001.2024-04.basic.20240520T151453Z.PROVISIONAL.zip could not be downloaded. URLs may have expired. Refreshing URL list. Refresh did not solve the isse. URL query for file NEON.D03.BARC.DP1.20033.001.2024-04.basic.20240520T151453Z.PROVISIONAL.zip failed. If all files fail, check data portal (data.neonscience.org/news) for possible outage alert. |======================================================================| 100% 1 files successfully downloaded to C:\Users\johns\Documents\GRIMe-AI\Downloads\NEON\Data/filesToStack20033 Backend QtAgg is interactive backend. Turning interactive mode on.

On Mon, Jul 29, 2024 at 7:10 PM John Stranzl @.***> wrote:

So I put the quotes around the True and False, and at the command prompt it asks if I want to download the data. So progress of a sort? But a) I don't want to be prompted and b) even though I said yes at the prompt, it did not download anything, still.

utils.capture_output(neonUtilities.zipsByProduct(dpID = strProduct, \ site = SiteCode, \ savepath = downloadsFilePath, \ startdate = strStartDate, \ enddate = strEndDate, \ package = 'basic', \ include_provisional = robjects.vectors.BoolVector(['True']), \ check_size = robjects.vectors.BoolVector(['False'])))

On Mon, Jul 29, 2024 at 7:00 PM John Stranzl @.***> wrote:

zipByProducts didn't download anything even though it didn't crash. I asked for Nitrate data from March 2024 through July 2024 and it did not download anything. I changed the range from March 2024 to June 2024 (because there's no data available for July) and it still did not download anything.

In the past few months, what has changed? This code was working find for me for 2+ years? Everything seemed to go wrong when NEON started marking more and more data provisional.

I am using this for my dissertation work. So you can imagine how this is putting a cramp in my work. :(

-John

On Mon, Jul 29, 2024 at 6:50 PM John Stranzl @.***> wrote:

Well... It no longer crashes but it doesn't download any data. I will see if I can get you any additional information. Here are the code changes I made based upon your previous email. I also had to change 'check_size' as shown in the following code.

utils.capture_output(neonUtilities.zipsByProduct(dpID = strProduct, \ site = SiteCode, \ savepath = downloadsFilePath, \ startdate = strStartDate, \ enddate = strEndDate, \ package = 'basic', \ include_provisional = robjects.vectors.BoolVector([True]), \ check_size = robjects.vectors.BoolVector([False])))

On Mon, Jul 29, 2024 at 2:03 PM Claire Lunch @.***> wrote:

@JohnStranzl https://github.com/JohnStranzl Thanks for the additional context, I was forgetting the issues around transferring boolean values between R and Python with rpy2. Unfortunately quotes work in some situations and not others, it depends on the details of the code on the R end. This runs successfully for me:

neonUtilities.zipsByProduct(dpID = strProduct, site = SiteCode, savepath = downloadsFilePath, startdate = strStartDate, enddate = strEndDate, package = 'basic', include_provisional = robjects.vectors.BoolVector([True]), check_size = 'FALSE')

We're currently working toward a Python version of neonUtilities, aiming for release to the public this fall - once that is available, you'll be able to code directly in Python and avoid rpy2 entirely! Keep an eye on Data Notifications https://www.neonscience.org/data-samples/data-notifications for the announcement, or you can sign up for Data Notification emails with a NEON user account https://www.neonscience.org/about/user-accounts.

Some background info on opting in for Provisional data here https://www.neonscience.org/impact/observatory-blog/change-default-neon-data-download-behavior .

— Reply to this email directly, view it on GitHub https://github.com/NEONScience/NEON-utilities/issues/133#issuecomment-2256583673, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIRROUV2QQTT24BEOEXNQDZOZ7XJAVCNFSM6AAAAABLTJR4JKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJWGU4DGNRXGM . You are receiving this because you were mentioned.Message ID: @.***>

cklunch commented 1 month ago

@JohnStranzl I can see you've wrapped zipsByProduct() inside a function. Have you tried running it independently, to isolate the error? Try running this, outside any other function:

neonUtilities.zipsByProduct(dpID='DP1.10026.001',
                            site='DEJU',
                            savepath='FILEPATH ON YOUR COMPUTER',
                            package='basic',
                            check_size=robjects.vectors.BoolVector([False]),
                            include_provisional=robjects.vectors.BoolVector([True]))

If that works, we can narrow down the problem to something in the wrapper code.

The "do you want to proceed" question is the size check, it's removed by check_size=robjects.vectors.BoolVector([False]).

The error messages in your latest comment indicate the files are found but download is failing. This happens most commonly because of a timeout. To increase the timeout in R, via rpy2, run base.options(timeout=300). The timeout is in seconds, so that will increase it to 5 minutes.

JohnStranzl commented 1 month ago

So... Here is the error that started me using 'include_provisional'

Finding available files |======================================================================| 100% Backend QtAgg is interactive backend. Turning interactive mode on. R[write to console]: Error in getZipUrls(month.urls, avg = avg, package = package, dpID = dpID, : All files found were provisional. Modify download query (dates and/or sites) or, if you want to use provisional data, use input parameter include.provisional=TRUE. R[write to console]: In addition: R[write to console]: Warning messages: R[write to console]: 1: R[write to console]: In strptime(x, fmt, tz = "GMT") : R[write to console]: unknown timezone 'America/New_York' R[write to console]: 2: R[write to console]: In strptime(x, fmt, tz = "GMT") : R[write to console]: unknown timezone 'GMT' R[write to console]: 3: R[write to console]: In strptime(x, fmt, tz = "GMT") : R[write to console]: unknown timezone 'America/New_York'

And everything has gone downhill from here.

So now with the R/python interface issue resolved with the solution you provided, there's no crash but nothing downloads.

I'll wait to hear from you at this point. I've blown up your email with all sorts of testing results.

On Mon, Jul 29, 2024 at 9:06 AM Claire Lunch @.***> wrote:

@JohnStranzl https://github.com/JohnStranzl TRUE and FALSE shouldn't be quoted, those are pre-defined terms in R. Try:

utils.capture_output(neonUtilities.zipsByProduct(dpID = strProduct, site = SiteCode, savepath = downloadsFilePath, startdate = strStartDate, enddate = strEndDate, package = 'basic', include_provisional = TRUE, check_size = FALSE))

Let us know if that resolves the error. Thank you!

— Reply to this email directly, view it on GitHub https://github.com/NEONScience/NEON-utilities/issues/133#issuecomment-2255887458, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIRROXTHLNYA5MVAWES7RDZOY44RAVCNFSM6AAAAABLTJR4JKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJVHA4DONBVHA . You are receiving this because you were mentioned.Message ID: @.***>

JohnStranzl commented 1 month ago

I will try these suggestions. Thanks. FYI... I am using the example provided on the NEON site and it has zipsbyproducts inside utils.capture_output which has worked fine for the past 2+ years.

On Mon, Jul 29, 2024 at 7:33 PM Claire Lunch @.***> wrote:

@JohnStranzl https://github.com/JohnStranzl I can see you've wrapped zipsByProduct() inside a function. Have you tried running it independently, to isolate the error? Try running this, outside any other function:

neonUtilities.zipsByProduct(dpID='DP1.10026.001', site='DEJU', savepath='FILEPATH ON YOUR COMPUTER', package='basic', check_size=robjects.vectors.BoolVector([False]), include_provisional=robjects.vectors.BoolVector([True]))

If that works, we can narrow down the problem to something in the wrapper code.

The "do you want to proceed" question is the size check, it's removed by check_size=robjects.vectors.BoolVector([False]).

The error messages in your latest comment indicate the files are found but download is failing. This happens most commonly because of a timeout. To increase the timeout in R, via rpy2, run base.options(timeout=300). The timeout is in seconds, so that will increase it to 5 minutes.

— Reply to this email directly, view it on GitHub https://github.com/NEONScience/NEON-utilities/issues/133#issuecomment-2257182857, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIRROVJ5SP3OLPKK7AJL63ZO3GMRAVCNFSM6AAAAABLTJR4JKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJXGE4DEOBVG4 . You are receiving this because you were mentioned.Message ID: @.***>

JohnStranzl commented 1 month ago

Increasing timeout and NOT calling zipsbyProduct inside the utils.capture_output functios did not work.

Finding available files |======================================================================| 100% Continuing will download files totaling approximately 3.25421 MB. Do you want to proceed y/n: >? y Downloading 2 files |======================================================================| 100% NEON.D03.BARC.DP1.20016.001.2024-03.basic.20240502T161350Z.PROVISIONAL.zip could not be downloaded. Re-attempting. NEON.D03.BARC.DP1.20016.001.2024-03.basic.20240502T161350Z.PROVISIONAL.zip could not be downloaded. URLs may have expired. Refreshing URL list. Refresh did not solve the isse. URL query for file NEON.D03.BARC.DP1.20016.001.2024-03.basic.20240502T161350Z.PROVISIONAL.zip failed. If all files fail, check data portal (data.neonscience.org/news) for possible outage alert. If file sizes are large, increase the timeout limit on your machine: options(timeout=###) NEON.D03.BARC.DP1.20016.001.2024-04.basic.20240520T151840Z.PROVISIONAL.zip could not be downloaded. Re-attempting.

neonUtilities = importr('neonUtilities') utils = importr('utils') base = importr('base')

base.options(timeout=3000)

try: ''' utils.capture_output(neonUtilities.zipsByProduct(dpID = strProduct, \

                                                 site = SiteCode, \
                                                 savepath =

downloadsFilePath, \ startdate = strStartDate, \ enddate = strEndDate, \ package = 'basic', \

include_provisional = robjects.vectors.BoolVector(['True']), \ load = robjects.vectors.BoolVector(['True']), \ check_size = robjects.vectors.BoolVector(['False'])))

'''
neon_zips_output = neonUtilities.zipsByProduct(dpID = strProduct, \
                                                 site = SiteCode, \
                                                 savepath =

downloadsFilePath, \ startdate = strStartDate, \ enddate = strEndDate, \ package = 'basic', \

include_provisional = robjects.vectors.BoolVector(['True']), \ load = robjects.vectors.BoolVector(['True']), \ check_size = robjects.vectors.BoolVector(['False']))

utils.capture_output(neon_zips_output)

On Mon, Jul 29, 2024 at 7:33 PM Claire Lunch @.***> wrote:

@JohnStranzl https://github.com/JohnStranzl I can see you've wrapped zipsByProduct() inside a function. Have you tried running it independently, to isolate the error? Try running this, outside any other function:

neonUtilities.zipsByProduct(dpID='DP1.10026.001', site='DEJU', savepath='FILEPATH ON YOUR COMPUTER', package='basic', check_size=robjects.vectors.BoolVector([False]), include_provisional=robjects.vectors.BoolVector([True]))

If that works, we can narrow down the problem to something in the wrapper code.

The "do you want to proceed" question is the size check, it's removed by check_size=robjects.vectors.BoolVector([False]).

The error messages in your latest comment indicate the files are found but download is failing. This happens most commonly because of a timeout. To increase the timeout in R, via rpy2, run base.options(timeout=300). The timeout is in seconds, so that will increase it to 5 minutes.

— Reply to this email directly, view it on GitHub https://github.com/NEONScience/NEON-utilities/issues/133#issuecomment-2257182857, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIRROVJ5SP3OLPKK7AJL63ZO3GMRAVCNFSM6AAAAABLTJR4JKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJXGE4DEOBVG4 . You are receiving this because you were mentioned.Message ID: @.***>

cklunch commented 1 month ago

@JohnStranzl Thanks for the update. A couple of details, and another thing to try-

  1. True and False shouldn't be quoted in robjects.vectors.BoolVector([True]), but I'm pretty sure that's unrelated to the download failure you're seeing now.
  2. Can you point me to the example code from NEON you've been following?
  3. The next possible solution to try is version conflicts. Can you update your version of R, and all R packages? If neonUtilities is the only R package you're using, the easiest way to do this is to update to the latest version of R and then re-install neonUtilities, it will install all the needed dependencies.
JohnStranzl commented 1 month ago

Thx. I tried True/False with and without quotes just in case.

I have tried different versions of R up to 4.2.2. I will have to try 4.4 when I get home from work today.

I will send the NEON example if I can still find it. I basically cut and past from a NEON example from 3 years ago.

On Tue, Jul 30, 2024, 9:31 AM Claire Lunch @.***> wrote:

@JohnStranzl https://github.com/JohnStranzl Thanks for the update. A couple of details, and another thing to try-

  1. True and False shouldn't be quoted in robjects.vectors.BoolVector([True]), but I'm pretty sure that's unrelated to the download failure you're seeing now.
  2. Can you point me to the example code from NEON you've been following?
  3. The next possible solution to try is version conflicts. Can you update your version of R, and all R packages? If neonUtilities is the only R package you're using, the easiest way to do this is to update to the latest version of R and then re-install neonUtilities, it will install all the needed dependencies.

— Reply to this email directly, view it on GitHub https://github.com/NEONScience/NEON-utilities/issues/133#issuecomment-2258360911, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIRROQGX2LMEP465YQBKWDZO6IUDAVCNFSM6AAAAABLTJR4JKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJYGM3DAOJRGE . You are receiving this because you were mentioned.Message ID: @.***>

JohnStranzl commented 1 month ago
  1. Removed quotes from TRUE and FALSE
  2. Upgraded to R-4.4.1-win
  3. Re-installed neonUtilities
  4. As for the sample code I am using, it can be found on this link: https://www.neonscience.org/resources/learning-hub/tutorials/neon-utilities-python .

I added the 'include_provisional' once it started telling me it couldn't download the data marked provisional without the additional parameter.

After making the above changes, the output remains the same. Says the URL expired. Timeout is set to 3000 because 300 didn't work so I figured, let's make it a huge timeout.

Downloading files totaling approximately 3.334567 MB Downloading 2 files |======================================================================| 100% NEON.D03.BARC.DP1.20016.001.2024-04.basic.20240520T151840Z.PROVISIONAL.zip could not be downloaded. Re-attempting. NEON.D03.BARC.DP1.20016.001.2024-04.basic.20240520T151840Z.PROVISIONAL.zip could not be downloaded. URLs may have expired. Refreshing URL list. Refresh did not solve the isse. URL query for file NEON.D03.BARC.DP1.20016.001.2024-04.basic.20240520T151840Z.PROVISIONAL.zip failed. If all files fail, check data portal (data.neonscience.org/news) for possible outage alert. If file sizes are large, increase the timeout limit on your machine: options(timeout=###) NEON.D03.BARC.DP1.20016.001.2024-05.basic.20240706T163346Z.PROVISIONAL.zip could not be downloaded. Re-attempting. NEON.D03.BARC.DP1.20016.001.2024-05.basic.20240706T163346Z.PROVISIONAL.zip could not be downloaded. URLs may have expired. Refreshing URL list. Refresh did not solve the isse. URL query for file NEON.D03.BARC.DP1.20016.001.2024-05.basic.20240706T163346Z.PROVISIONAL.zip failed. If all files fail, check data portal (data.neonscience.org/news) for possible outage alert. If file sizes are large, increase the timeout limit on your machine: options(timeout=###) 2 files successfully downloaded to C:\Users\johns\Documents\GRIMe-AI\Downloads\NEON\Data/filesToStack20016

On Tue, Jul 30, 2024 at 9:31 AM Claire Lunch @.***> wrote:

@JohnStranzl https://github.com/JohnStranzl Thanks for the update. A couple of details, and another thing to try-

  1. True and False shouldn't be quoted in robjects.vectors.BoolVector([True]), but I'm pretty sure that's unrelated to the download failure you're seeing now.
  2. Can you point me to the example code from NEON you've been following?
  3. The next possible solution to try is version conflicts. Can you update your version of R, and all R packages? If neonUtilities is the only R package you're using, the easiest way to do this is to update to the latest version of R and then re-install neonUtilities, it will install all the needed dependencies.

— Reply to this email directly, view it on GitHub https://github.com/NEONScience/NEON-utilities/issues/133#issuecomment-2258360911, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIRROQGX2LMEP465YQBKWDZO6IUDAVCNFSM6AAAAABLTJR4JKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJYGM3DAOJRGE . You are receiving this because you were mentioned.Message ID: @.***>

JohnStranzl commented 1 month ago

You had asked, "What if I ran it at the console?" Here is the result if I omit the provisional parameter: neon_zips = neonUtilities.zipsByProduct(dpID=strProduct, \ site=base.c(SiteCode), \ savepath=downloadsFilePath, \ startdate=strStartDate, \ enddate=strEndDate, \ package='basic'); Finding available files |======================================================================| 100% R[write to console]: Error in getZipUrls(month.urls, avg = avg, package = package, dpID = dpID, : All files found were provisional. Modify download query (dates and/or sites) or, if you want to use provisional data, use input parameter include.provisional=TRUE.

On Tue, Jul 30, 2024 at 9:39 PM John E Stranzl Jr @.***> wrote:

  1. Removed quotes from TRUE and FALSE
  2. Upgraded to R-4.4.1-win
  3. Re-installed neonUtilities
  4. As for the sample code I am using, it can be found on this link:

https://www.neonscience.org/resources/learning-hub/tutorials/neon-utilities-python .

I added the 'include_provisional' once it started telling me it couldn't download the data marked provisional without the additional parameter.

After making the above changes, the output remains the same. Says the URL expired. Timeout is set to 3000 because 300 didn't work so I figured, let's make it a huge timeout.

Downloading files totaling approximately 3.334567 MB Downloading 2 files |======================================================================| 100% NEON.D03.BARC.DP1.20016.001.2024-04.basic.20240520T151840Z.PROVISIONAL.zip could not be downloaded. Re-attempting. NEON.D03.BARC.DP1.20016.001.2024-04.basic.20240520T151840Z.PROVISIONAL.zip could not be downloaded. URLs may have expired. Refreshing URL list. Refresh did not solve the isse. URL query for file NEON.D03.BARC.DP1.20016.001.2024-04.basic.20240520T151840Z.PROVISIONAL.zip failed. If all files fail, check data portal (data.neonscience.org/news) for possible outage alert. If file sizes are large, increase the timeout limit on your machine: options(timeout=###) NEON.D03.BARC.DP1.20016.001.2024-05.basic.20240706T163346Z.PROVISIONAL.zip could not be downloaded. Re-attempting. NEON.D03.BARC.DP1.20016.001.2024-05.basic.20240706T163346Z.PROVISIONAL.zip could not be downloaded. URLs may have expired. Refreshing URL list. Refresh did not solve the isse. URL query for file NEON.D03.BARC.DP1.20016.001.2024-05.basic.20240706T163346Z.PROVISIONAL.zip failed. If all files fail, check data portal (data.neonscience.org/news) for possible outage alert. If file sizes are large, increase the timeout limit on your machine: options(timeout=###) 2 files successfully downloaded to C:\Users\johns\Documents\GRIMe-AI\Downloads\NEON\Data/filesToStack20016

On Tue, Jul 30, 2024 at 9:31 AM Claire Lunch @.***> wrote:

@JohnStranzl https://github.com/JohnStranzl Thanks for the update. A couple of details, and another thing to try-

  1. True and False shouldn't be quoted in robjects.vectors.BoolVector([True]), but I'm pretty sure that's unrelated to the download failure you're seeing now.
  2. Can you point me to the example code from NEON you've been following?
  3. The next possible solution to try is version conflicts. Can you update your version of R, and all R packages? If neonUtilities is the only R package you're using, the easiest way to do this is to update to the latest version of R and then re-install neonUtilities, it will install all the needed dependencies.

— Reply to this email directly, view it on GitHub < https://github.com/NEONScience/NEON-utilities/issues/133#issuecomment-2258360911>,

or unsubscribe < https://github.com/notifications/unsubscribe-auth/ADIRROQGX2LMEP465YQBKWDZO6IUDAVCNFSM6AAAAABLTJR4JKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJYGM3DAOJRGE>

. You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/NEONScience/NEON-utilities/issues/133#issuecomment-2259468927, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIRROXXTLAPCB4OCRXECTLZPA53RAVCNFSM6AAAAABLTJR4JKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJZGQ3DQOJSG4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

cklunch commented 1 month ago

@JohnStranzl Right, sorry if this was unclear, there are multiple things going on - the failure to download has nothing to do with the include_provisional parameter, we've resolved that problem with the include_provisional = robjects.vectors.BoolVector([True]) input. Without that input, the function doesn't even attempt to download provisional data.

To figure out why the downloads are failing, the next possibility is a firewall issue, either on your machine or on your network. If you haven't already, try downloading from a different computer and from a different network.

If neither of those make a difference, I think it would be easiest to troubleshoot in real time. If you're up for getting on a zoom call, can you submit a question through the NEON Contact Us form, and let's set up a time.

JohnStranzl commented 1 month ago

I un-installed rpy2, neonUtilities, all versions of R, restarted from scratch, and I have it working. I have not gone back through the release notes to see what changed since February that would cause this massive issue after having no issues for 3 years or so.

The remaining issue that it is looking for R_HOME in the os environment variables which, until now, I was able to add programmatically from my python script. Now, it has to be manually added to the os environment variables. Any ideas on this issue? If like to keep it programmatic so those less versed in Windows internals do not have to worry about this.

BTW... Thanks for all your support.

On Wed, Jul 31, 2024, 9:30 AM Claire Lunch @.***> wrote:

@JohnStranzl https://github.com/JohnStranzl Right, sorry if this was unclear, there are multiple things going on - the failure to download has nothing to do with the include_provisional parameter, we've resolved that problem with the include_provisional = robjects.vectors.BoolVector([True]) input. Without that input, the function doesn't even attempt to download provisional data.

To figure out why the downloads are failing, the next possibility is a firewall issue, either on your machine or on your network. If you haven't already, try downloading from a different computer and from a different network.

If neither of those make a difference, I think it would be easiest to troubleshoot in real time. If you're up for getting on a zoom call, can you submit a question through the NEON Contact Us https://www.neonscience.org/about/contact-us form, and let's set up a time.

— Reply to this email directly, view it on GitHub https://github.com/NEONScience/NEON-utilities/issues/133#issuecomment-2260531098, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIRROT6AK47LAPGKHIO42LZPDRFRAVCNFSM6AAAAABLTJR4JKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENRQGUZTCMBZHA . You are receiving this because you were mentioned.Message ID: @.***>

cklunch commented 1 month ago

@JohnStranzl That's great news, I'm so glad to hear it's working now! Based on what fixed it, it was most likely a version conflict in the packages neonUtilities depends on.

The R_HOME issue sounds like something in the rpy2 settings, possibly they've changed the default behavior. You should be able to set an os environment variable programmatically, though.

Like I said before, stay tuned for the Python version of neonUtilities - at least then you'll only be managing dependencies in one language!