search

DataONEorg / rdataone

R package for reading and writing data at DataONE data repositories
http://doi.org/10.5063/F1M61H5X
36 stars 19 forks source link

getDataPackage() fails with "subscript out of bounds" #243

Closed csjx closed 4 years ago

csjx commented 4 years ago

Given the following code to download a data package:

library(dataone)
library(datapack)
d1c <- D1Client("PROD", "urn:node:ARCTIC")
mn <- d1c@mn
packageId <- "resource_map_urn:uuid:7c87b906-adfb-4f02-b07a-4a1c344e1118"
pkg <- getDataPackage(d1c, identifier=packageId, lazyLoad=TRUE, limit="0MB", quiet=FALSE)

it fails with:

Error accessing https://arcticdata.io/metacat/d1/mn/v2/query/solr/q=id:%22resource_map_urn:uuid:7c87b906-adfb-4f02-b07a-4a1c344e1118%22&fl=isDocumentedBy,resourceMap,documents,formatType: subscript out of bounds

Error in .local(x, identifier, ...) : 
  Unable to get response for identifier resource_map_urn:uuid:7c87b906-adfb-4f02-b07a-4a1c344e1118

whereas the direct browser URL succeeds: https://arcticdata.io/metacat/d1/mn/v2/query/solr/q=id:%22resource_map_urn:uuid:7c87b906-adfb-4f02-b07a-4a1c344e1118%22&fl=isDocumentedBy,resourceMap,documents,formatType

<?xml version="1.0" encoding="UTF-8"?>
<response>
    <lst name="responseHeader">
        <int name="status">0</int>
        <int name="QTime">77</int>
        <lst name="params">
            <str name="fq">(readPermission:"public")OR(isPublic:true)</str>
            <str name="fl">isDocumentedBy,resourceMap,documents,formatType</str>
            <str name="q">id:"resource_map_urn:uuid:7c87b906-adfb-4f02-b07a-4a1c344e1118"</str>
        </lst>
    </lst>
    <result name="response" numFound="1" start="0">
        <doc>
            <str name="formatType">RESOURCE</str>
        </doc>
    </result>
</response>

Perhaps the out-of-bounds error occurs since some of the desired attributes are not present in the result?

I've tested this against the ARCTIC and PISCO nodes with the same results with different identifiers known to be in the Solr index.

gothub commented 4 years ago

@csjx what version of dataone R package are you using? I entered your R commands for ARCTIC above using v2.1.3 and it worked.

ranicrab commented 4 years ago

Hi! I checked and I am using v2.1.3 (not sure what Chris is using). I also got a similar error when I tried to load a new package on the PISCO node:

packageId <- uploadDataPackage(d1c, dp, public = TRUE) Error in getNodeSet(xml, "/*[local-name()='identifier']")[[1]] : subscript out of bounds

gothub commented 4 years ago

@ranicrab very interesting, thx for the report. Could you send me the entire script that you are using? I'd like to try to recreate this on a 'DEV' environment node.

@csjx if you have a minute, could you 'clear' your R environment and run just the lines you send me (after checking the R version). I'm still unable to recreate the error you posted.

csjx commented 4 years ago

@gothub @ranicrab - So yeah, after clearing my environment and starting a new R session, the code runs fine for me now too. I'm using v2.1.3 too.

gothub commented 4 years ago

@csjx my concern here is that something you may have set in your R environment caused the failure (i.e. R variables not being initialized correctly inside dataone). If you do run into this again, pls send your R environment (Rstudio -> Environment -> Global environment). Thanks for checking this!

ranicrab commented 4 years ago

Hi Peter,

This is my first github posting etc. so if I was supposed to put the file directly on there let me know - anyway, I have attached my history and data file (I got the same error this morning - different machine, new session):

Error in getNodeSet(xml, "/*[local-name()='identifier']")[[1]] : subscript out of bounds

I am also attaching the files I am using, so that you can reproduce fully - On 2020-02-07 9:51 AM, Peter Slaughter wrote:

@ranicrab https://github.com/ranicrab very interesting. Could you send me the entire script that you are using? I'd like to try to recreate this on a 'DEV' environment node.

@csjx https://github.com/csjx if you have a minute, could you 'clear' your R environment and run just the lines you send me (after checking the R version). I'm still unable to recreate the error you posted.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DataONEorg/rdataone/issues/243?email_source=notifications&email_token=AOOR4G26ZRQVQPRJDKIRDMLRBWNSBA5CNFSM4KRGL34KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELD6MSA#issuecomment-583525960, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOOR4G3LTF4V3THMUXIXXJLRBWNSBANCNFSM4KRGL34A.

library(dataone) library(datapack) library(EML) d1c<-D1Client("PROD","urn:node:PISCO") mn <- d1c@mn dp <- new("DataPackage") dp metadata_id <- "doi:10.6085/AA/KHLX00_XXXITV2XLSR01_20190220.50.1" data_id <- "doi:10.6085/AA/KHLX00_XXXITV2XLSR01_20190220.40.1" metadataObj <- new("DataObject", id=metadata_id, format="eml://ecoinformatics.org/eml-2.0.1", filename="KHLX00_XXXITV2XLSR01_20190220.50.1") metadataObj <- new("DataObject", id=metadata_id, format="eml://ecoinformatics.org/eml-2.0.1", filename="KHLX00_XXXITV2XLSR01_20190220.50.1") dp <- addMember(dp, metadataObj) dp sourceObj <- new("DataObject", id=data_id, format="text/csv", filename="KHLX00_XXXITV2XLSR01_20190220.40.1") dp <- addMember(dp, sourceObj, mo=metadataObj) dp packageId <- uploadDataPackage(d1c, dp, public = TRUE) savehistory("~/work/R/loaddata.Rhistory")

date time yearday tempc flag 2019-02-20 03:10:00.00Z 50.13194444 9.16 00000 ...

gothub commented 4 years ago

@ranicrab thx for sending your script. I was able to use the same commands, substituting my own pids and data files, to dev.nceas - so I'm not sure yet what is causing the problem. If you have a minute - could you send your R environment to slaughter@nceas.ucsb.edu?

In the Rstudio console, you can type 'ctrl-l' (that's lowercase L) and then type 'print(as.list(.GlobalEnv))'. If you could cut/paste that in an email, that would help a lot, even if the output is many lines.

ranicrab commented 4 years ago

Here you go!

$metadataObj Access identifer : doi:10.6085/AA/KHLX00_XXXITV2XLSR01_20190220.50.1 submitter : NA rightHolder : NA access policy: No access policy defined Physical formatId : eml://ecoinformatics.org/eml-2.0.1 mediaType : NA size : 15886 System seriesId : NA serialVersion : 1 obsoletes : NA obsoletedBy : NA archived : FALSE dateUploaded : NA dateSysMetadataModified : 2020-02-07T19:13:21Z Data filename : C:\Users\Dragonoak\Documents\work\R\KHLX00_XXXITV2XLSR01_20190220.50.1 $dp Members: filename format mediaType size identifier modified local KHL...40.1 text/csv NA 1540647 doi...40.1 n y KHL...50.1 em...0.1 NA 15886 doi...50.1 n y Package identifier: NA RightsHolder: NA Relationships (updated): subject predicate object 2 KHLX0...0.40.1 cito:...ntedBy KHLX0...0.50.1 1 KHLX0...0.50.1 cito:documents KHLX0...0.40.1 $metadata_id [1] "doi:10.6085/AA/KHLX00_XXXITV2XLSR01_20190220.50.1" $d1c An object of class "D1Client" Slot "cn": An object of class "CNode" Slot "endpoint": [1] "https://cn.dataone.org/cn/v2" Slot "identifier": [1] "urn:node:CNUCSB1" Slot "name": [1] "cn-ucsb-1" Slot "description": [1] "University of California Santa Barbara Coordinating Node" Slot "baseURL": [1] "https://cn.dataone.org/cn" Slot "subject": [1] "CN=urn:node:CNUCSB1,DC=dataone,DC=org" Slot "contactSubject": [1] "CN=Robert P Waltz A904,O=Google,C=US,DC=cilogon,DC=org" Slot "replicate": [1] "false" Slot "type": [1] "cn" Slot "state": [1] "up" Slot "services": name version available 1 CNCore v1 true 2 CNRead v1 true 3 CNAuthorization v1 true 4 CNReplication v1 true 5 CNRegister v1 true 6 CNCore v2 true 7 CNRead v2 true 8 CNAuthorization v2 true 9 CNIdentity v2 true 10 CNReplication v2 true 11 CNRegister v2 true 12 CNDiagnostic v2 true 13 CNView v2 true Slot "serviceUrls": service 1 query.solr Url 1 https://cn.dataone.org/cn/v2/query/solr/? Slot "APIversion": [1] "v2" Slot "env": [1] "prod" Slot "mn": An object of class "MNode" Slot "endpoint": [1] "http://data.piscoweb.org/catalog/d1/mn/v2" Slot "identifier": [1] "urn:node:PISCO" Slot "name": [1] "PISCO MN" Slot "description": [1] "The Partnership for Interdisciplinary Studies of Coastal Oceans is a long-term ecosystem research and monitoring program established with the goals of understanding dynamics of the coastal ocean ecosystem along the U.S. west coast, sharing that knowledge so ocean managers and policy makers can take science&#8208" Slot "baseURL": [1] "http://data.piscoweb.org/catalog/d1/mn" Slot "subject": [1] "CN=urn:node:PISCO,DC=dataone,DC=org" Slot "contactSubject": [1] "CN=Michael Frenock A5618,O=Google,C=US,DC=cilogon,DC=org" Slot "replicate": [1] "false" Slot "type": [1] "mn" Slot "state": [1] "up" Slot "services": name version available 1 MNCore v1 true 2 MNCore v2 true 3 MNRead v1 true 4 MNRead v2 true 5 MNAuthorization v1 true 6 MNAuthorization v2 true 7 MNStorage v1 true 8 MNStorage v2 true 9 MNReplication v1 true 10 MNReplication v2 true 11 MNPackage v1 true 12 MNPackage v2 true 13 MNView v2 true Slot "serviceUrls": service 1 query.solr Url 1 http://data.piscoweb.org/catalog/d1/mn/v2/query/solr/ Slot "APIversion": [1] "v2" Slot "env": [1] "prod" $sourceObj Access identifer : doi:10.6085/AA/KHLX00_XXXITV2XLSR01_20190220.40.1 submitter : NA rightHolder : NA access policy: No access policy defined Physical formatId : text/csv mediaType : NA size : 1540647 System seriesId : NA serialVersion : 1 obsoletes : NA obsoletedBy : NA archived : FALSE dateUploaded : NA dateSysMetadataModified : 2020-02-07T19:13:58Z Data filename : C:\Users\Dragonoak\Documents\work\R\KHLX00_XXXITV2XLSR01_20190220.40.1 $data_id [1] "doi:10.6085/AA/KHLX00_XXXITV2XLSR01_20190220.40.1" $mn An object of class "MNode" Slot "endpoint": [1] "http://data.piscoweb.org/catalog/d1/mn/v2" Slot "identifier": [1] "urn:node:PISCO" Slot "name": [1] "PISCO MN" Slot "description": [1] "The Partnership for Interdisciplinary Studies of Coastal Oceans is a long-term ecosystem research and monitoring program established with the goals of understanding dynamics of the coastal ocean ecosystem along the U.S. west coast, sharing that knowledge so ocean managers and policy makers can take science&#8208" Slot "baseURL": [1] "http://data.piscoweb.org/catalog/d1/mn" Slot "subject": [1] "CN=urn:node:PISCO,DC=dataone,DC=org" Slot "contactSubject": [1] "CN=Michael Frenock A5618,O=Google,C=US,DC=cilogon,DC=org" Slot "replicate": [1] "false" Slot "type": [1] "mn" Slot "state": [1] "up" Slot "services": name version available 1 MNCore v1 true 2 MNCore v2 true 3 MNRead v1 true 4 MNRead v2 true 5 MNAuthorization v1 true 6 MNAuthorization v2 true 7 MNStorage v1 true 8 MNStorage v2 true 9 MNReplication v1 true 10 MNReplication v2 true 11 MNPackage v1 true 12 MNPackage v2 true 13 MNView v2 true Slot "serviceUrls": service 1 query.solr Url 1 http://data.piscoweb.org/catalog/d1/mn/v2/query/solr/ Slot "APIversion": [1] "v2" Slot "env": [1] "prod"

On 2020-02-07 12:18 PM, Peter Slaughter wrote:

@ranicrab https://github.com/ranicrab thx for sending your script. I was able to use the same commands, substituting my own pids and data files, to dev.nceas - so I'm not sure yet what is causing the problem. If you have a minute - could you send your R environment to slaughter@nceas.ucsb.edu mailto:slaughter@nceas.ucsb.edu?

In the Rstudio console, you can type 'ctrl-l' (that's lowercase L) and then type 'print(as.list(.GlobalEnv))'. If you could cut/paste that in an email, that would help a lot, even if the output is many lines.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DataONEorg/rdataone/issues/243?email_source=notifications&email_token=AOOR4G3F3SU6Q66VQWXZHL3RBW6YZA5CNFSM4KRGL34KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELEO4IQ#issuecomment-583593506, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOOR4G6RPXAMTMUQVNHFZITRBW6YZANCNFSM4KRGL34A.

-- Rani Gaddam Research Associate and Data Manager MARINe Research Group at UC Santa Cruz Long Marine Lab 115 McAllister Way, Santa Cruz,Ca 95060 (831) 459-1621 office (831) 459-3383 fax gaddam@ucsc.edu http://pacificrockyintertidal.org http://intertidalmap.org

gothub commented 4 years ago

@ranicrab thx for the env list - I just looked at this in detail and it all looks fine. I'll keep digging...

gothub commented 4 years ago

@ranicrab since I'm unable to reproduce the error you posted, if you could try:

d1c<-D1Client("PROD","urn:node:PISCO")
nodes <- listNodes(d1c@cn)

... and if that works without error, then try (provided you have a current authentication token):

reserveIdentifier(d1c@cn, "urn:uuid:397082B7-B039-42E5-A75C-D1AE075ACF46")

These are the only two functions that could produce the error that you listed. Thanks a lot for trying this, this is a big help.

ranicrab commented 4 years ago

Hi! I tried those commands and didn't get any errors:

d1c<-D1Client("PROD","urn:node:PISCO") nodes <- listNodes(d1c@cn) reserveIdentifier(d1c@cn, "urn:uuid:397082B7-B039-42E5-A75C-D1AE075ACF46") [1] "urn:uuid:397082B7-B039-42E5-A75C-D1AE075ACF46"

Let me know if there is anything else I should do?

On 2020-02-07 2:14 PM, Peter Slaughter wrote:

@ranicrab https://github.com/ranicrab since I'm unable to reproduce the error you posted, if you could try:

|d1c<-D1Client("PROD","urn:node:PISCO") nodes <- listNodes(d1c@cn) |

... and if that works without error, then try (provided you have a current authentication token):

|reserveIdentifier(d1c@cn, "urn:uuid:397082B7-B039-42E5-A75C-D1AE075ACF46") |

These are the only two functions that could produce the error that you listed. Thanks a lot for trying this, this is a big help.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DataONEorg/rdataone/issues/243?email_source=notifications&email_token=AOOR4G6EWVKQ7MH2DZAG5DTRBXML5A5CNFSM4KRGL34KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELE2EGI#issuecomment-583639577, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOOR4GZLTTWIUFO2Q4YIQDLRBXML5ANCNFSM4KRGL34A.

-- Rani Gaddam Research Associate and Data Manager MARINe Research Group at UC Santa Cruz Long Marine Lab 115 McAllister Way, Santa Cruz,Ca 95060 (831) 459-1621 office (831) 459-3383 fax gaddam@ucsc.edu http://pacificrockyintertidal.org http://intertidalmap.org

gothub commented 4 years ago

@ranicrab let me know if you are still having uploadDataPackage issues. I'll improve the error checking for that function to prevent this error.

ranicrab commented 4 years ago

Hi - I tried again today and got the same error:

Error in getNodeSet(xml, "/*[local-name()='identifier']")[[1]] :   subscript out of bounds

Do you think this has anything to do with me not being in the PISCO data managers group? I don't know if I have been added or not - I don't know if it would matter since this is a new dataset but maybe I am unable to load data to PISCO?

OK - thanks!

-Rani

On 2020-02-10 1:20 PM, Peter Slaughter wrote:

@ranicrab https://github.com/ranicrab let me know if you are still having |uploadDataPackage| issues. I'll improve the error checking for that function to prevent this error.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DataONEorg/rdataone/issues/243?email_source=notifications&email_token=AOOR4G5CETRDTXY7S5MSQXDRCHAJNA5CNFSM4KRGL34KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELKKNCI#issuecomment-584361609, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOOR4G6ZVWIBEWR3QL5AX4DRCHAJNANCNFSM4KRGL34A.

-- Rani Gaddam Research Associate and Data Manager MARINe Research Group at UC Santa Cruz Long Marine Lab 115 McAllister Way, Santa Cruz,Ca 95060 (831) 459-1621 office (831) 459-3383 fax gaddam@ucsc.edu http://pacificrockyintertidal.org http://intertidalmap.org

gothub commented 4 years ago

Since you are uploading a new package (instead of updating an existing one) I would expect that you could upload it using your DataONE token, whether or not you belong to the data managers group. However, the two pids that you showed in your previous post (the env post) are not in DataONE:

https://data.piscoweb.org/catalog/d1/mn/v2/meta/doi:10.6085/AA/KHLX00_XXXITV2XLSR01_20190220.50.1
https://data.piscoweb.org/catalog/d1/mn/v2/meta/doi:10.6085/AA/KHLX00_XXXITV2XLSR01_20190220.40.1

Is your token current? You can check it by typing this command into your R console, for example:

> getTokenInfo(AuthenticationManager())
           name                              subject                 end expired
1 dataone_token http://orcid.org/0000-0002-2192-403X 2020-02-11 16:01:01   FALSE
ranicrab commented 4 years ago

Hi! My token is current (I updated it before trying to submit the package again), and I did run the check to make sure:

getTokenInfo(AuthenticationManager())            name                              subject 1 dataone_token http://orcid.org/0000-0003-1501-0861                   end expired 1 2020-02-11 15:38:25   FALSE

On 2020-02-10 2:03 PM, Peter Slaughter wrote:

Since you are uploading a new package (instead of updating an existing one) I would expect that you could upload it using your DataONE token, whether or not you belong to the data managers group. However, the two pids that you showed in your previous post (the env post) are not in DataONE:

https://data.piscoweb.org/catalog/d1/mn/v2/meta/doi:10.6085/AA/KHLX00_XXXITV2XLSR01_20190220.50.1 https://data.piscoweb.org/catalog/d1/mn/v2/meta/doi:10.6085/AA/KHLX00_XXXITV2XLSR01_20190220.40.1

Is your token current? You can check it by typing this command into your R console, for example:

|> getTokenInfo(AuthenticationManager()) name subject end expired 1 dataone_token http://orcid.org/0000-0002-2192-403X 2020-02-11 16:01:01 FALSE |

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DataONEorg/rdataone/issues/243?email_source=notifications&email_token=AOOR4G5M7ICVVACXHFX427LRCHFJXA5CNFSM4KRGL34KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELKO3IY#issuecomment-584379811, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOOR4G2WTBWGVKNJWCMTKPDRCHFJXANCNFSM4KRGL34A.

-- Rani Gaddam Research Associate and Data Manager MARINe Research Group at UC Santa Cruz Long Marine Lab 115 McAllister Way, Santa Cruz,Ca 95060 (831) 459-1621 office (831) 459-3383 fax gaddam@ucsc.edu http://pacificrockyintertidal.org http://intertidalmap.org

gothub commented 4 years ago

@ranicrab It looks like you are doing everything correctly. I'd like to talk to @csjx about this and then get back to you - probably by tomorrow mid-day.

gothub commented 4 years ago

@ranicrab As a test of the PISCO member node, I went to https://data.piscoweb.org/metacatui/#share and tried to log in (clicking "Sign In") and got some unintended text printed to the web page and was unable to login. There seems to be a problem with metacatui/metacat. Do you have an MN operator who could look into this?

Once this problem is resolved, I'd like to perform a test by uploading a single object to your MN, which will not appear in the search results.

BTW - How are you creating a user token?

ranicrab commented 4 years ago

I see that text as well and forwarded this message to Mike Frenock at OSU since he is the one who manages the PISCO/Metacat things.

I have been logging in here:

https://search.dataone.org/data

and then going to Profile > Settings to get my token to paste into R.

On 2020-02-11 10:20 AM, Peter Slaughter wrote:

@ranicrab https://github.com/ranicrab As a test of the PISCO member node, I went to https://data.piscoweb.org/metacatui/#share and tried to log in (clicking "Sign In") and got some unintended text printed to the web page and was unable to login. There seems to be a problem with metacatui/metacat. Do you have an MN operator who might want to look into this?

Once this problem is resolved, I'd like to perform a test by uploading a single object to your MN, which will not appear in the search results.

BTW - How are you creating a user token?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DataONEorg/rdataone/issues/243?email_source=notifications&email_token=AOOR4G7HTTYRFFEZZBHGTK3RCLUADA5CNFSM4KRGL34KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELNQHZI#issuecomment-584778725, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOOR4G6K2UT5LVJNA373TSTRCLUADANCNFSM4KRGL34A.

-- Rani Gaddam Research Associate and Data Manager MARINe Research Group at UC Santa Cruz Long Marine Lab 115 McAllister Way, Santa Cruz,Ca 95060 (831) 459-1621 office (831) 459-3383 fax gaddam@ucsc.edu http://pacificrockyintertidal.org http://intertidalmap.org

ranicrab commented 4 years ago

Hi Peter,

This problem is now resolved and I am able to login, though I can't view any data so I think there is still a problem (Mike wrote Jing about the problems he was having).

Let me know if there is anything else I can do to help/troubleshoot.

-Rani

On 2020-02-11 10:20 AM, Peter Slaughter wrote:

@ranicrab https://github.com/ranicrab As a test of the PISCO member node, I went to https://data.piscoweb.org/metacatui/#share and tried to log in (clicking "Sign In") and got some unintended text printed to the web page and was unable to login. There seems to be a problem with metacatui/metacat. Do you have an MN operator who might want to look into this?

Once this problem is resolved, I'd like to perform a test by uploading a single object to your MN, which will not appear in the search results.

BTW - How are you creating a user token?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DataONEorg/rdataone/issues/243?email_source=notifications&email_token=AOOR4G7HTTYRFFEZZBHGTK3RCLUADA5CNFSM4KRGL34KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELNQHZI#issuecomment-584778725, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOOR4G6K2UT5LVJNA373TSTRCLUADANCNFSM4KRGL34A.

-- Rani Gaddam Research Associate and Data Manager MARINe Research Group at UC Santa Cruz Long Marine Lab 115 McAllister Way, Santa Cruz,Ca 95060 (831) 459-1621 office (831) 459-3383 fax gaddam@ucsc.edu http://pacificrockyintertidal.org http://intertidalmap.org

ranicrab commented 4 years ago

Hi Peter,

Mike and Jing figured out the Metacat issues so things are working again on the PISCO end of things - however I just tried to load again using R and got the same error.

However now I think you can test on your end so hopefully you can figure it out!

Let me know if there is anything else I can do.

-Rani

On 2020-02-11 10:20 AM, Peter Slaughter wrote:

@ranicrab https://github.com/ranicrab As a test of the PISCO member node, I went to https://data.piscoweb.org/metacatui/#share and tried to log in (clicking "Sign In") and got some unintended text printed to the web page and was unable to login. There seems to be a problem with metacatui/metacat. Do you have an MN operator who might want to look into this?

Once this problem is resolved, I'd like to perform a test by uploading a single object to your MN, which will not appear in the search results.

BTW - How are you creating a user token?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DataONEorg/rdataone/issues/243?email_source=notifications&email_token=AOOR4G7HTTYRFFEZZBHGTK3RCLUADA5CNFSM4KRGL34KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELNQHZI#issuecomment-584778725, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOOR4G6K2UT5LVJNA373TSTRCLUADANCNFSM4KRGL34A.

-- Rani Gaddam Research Associate and Data Manager MARINe Research Group at UC Santa Cruz Long Marine Lab 115 McAllister Way, Santa Cruz,Ca 95060 (831) 459-1621 office (831) 459-3383 fax gaddam@ucsc.edu http://pacificrockyintertidal.org http://intertidalmap.org

ranicrab commented 4 years ago

Hi again,

Here is the error from the log - likely from when I tried to upload data? Just in case it helps!

Feb 12, 2020 2:42:05 PM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet [metacat] in context with path [/catalog] threw exception java.lang.IllegalStateException: getOutputStream() has already been called for this response  at org.apache.catalina.connector.Response.getWriter(Response.java:662)  at org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:213)  at edu.ucsb.nceas.metacat.MetaCatServlet.handleGetOrPost(MetaCatServlet.java:1243)  at edu.ucsb.nceas.metacat.MetaCatServlet.doPost(MetaCatServlet.java:507)  at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)  at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)  at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)  at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)  at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)  at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)  at java.lang.Thread.run(Thread.java:745)

On 2020-02-12 2:36 PM, Rani Gaddam wrote:

Hi Peter,

Mike and Jing figured out the Metacat issues so things are working again on the PISCO end of things - however I just tried to load again using R and got the same error.

However now I think you can test on your end so hopefully you can figure it out!

Let me know if there is anything else I can do.

-Rani

On 2020-02-11 10:20 AM, Peter Slaughter wrote:

@ranicrab https://github.com/ranicrab As a test of the PISCO member node, I went to https://data.piscoweb.org/metacatui/#share and tried to log in (clicking "Sign In") and got some unintended text printed to the web page and was unable to login. There seems to be a problem with metacatui/metacat. Do you have an MN operator who might want to look into this?

Once this problem is resolved, I'd like to perform a test by uploading a single object to your MN, which will not appear in the search results.

BTW - How are you creating a user token?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DataONEorg/rdataone/issues/243?email_source=notifications&email_token=AOOR4G7HTTYRFFEZZBHGTK3RCLUADA5CNFSM4KRGL34KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELNQHZI#issuecomment-584778725, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOOR4G6K2UT5LVJNA373TSTRCLUADANCNFSM4KRGL34A.

-- Rani Gaddam Research Associate and Data Manager MARINe Research Group at UC Santa Cruz Long Marine Lab 115 McAllister Way, Santa Cruz,Ca 95060 (831) 459-1621 office (831) 459-3383 fax gaddam@ucsc.edu http://pacificrockyintertidal.org http://intertidalmap.org

-- Rani Gaddam Research Associate and Data Manager MARINe Research Group at UC Santa Cruz Long Marine Lab 115 McAllister Way, Santa Cruz,Ca 95060 (831) 459-1621 office (831) 459-3383 fax gaddam@ucsc.edu http://pacificrockyintertidal.org http://intertidalmap.org

gothub commented 4 years ago

@ranicrab One of our developers at NCEAS identified the problem. After the URL for the PISCO member node was updated in DataONE (on the coordinating node) the problem has been resolved. Please restart your R session and attempt to upload your data package again.

gothub commented 4 years ago

The R client gets the MN URL from the CN. In the case of PISCO, the URL had to change from http://... to 'https://as http: was no longer supported. The MNodecreateObject()` call was updated so that it can recover from this type of error. This fix was made in commit 34884f8577cb834cecb00f1b3dcca183d582278b

ranicrab commented 4 years ago

Success! Thanks!

On 2020-02-12 4:06 PM, Peter Slaughter wrote:

@ranicrab https://github.com/ranicrab One of our developers at NCEAS identified the problem. After the URL for the PISCO member node was updated in DataONE (on the coordinating node) the problem has been resolved. Please restart your R session and attempt to upload your data package again.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DataONEorg/rdataone/issues/243?email_source=notifications&email_token=AOOR4GZTEKKUHQJEXBPCI7TRCSFKDA5CNFSM4KRGL34KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELS32JI#issuecomment-585481509, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOOR4GZN7IPCGIL47GMNJH3RCSFKDANCNFSM4KRGL34A.

-- Rani Gaddam Research Associate and Data Manager MARINe Research Group at UC Santa Cruz Long Marine Lab 115 McAllister Way, Santa Cruz,Ca 95060 (831) 459-1621 office (831) 459-3383 fax gaddam@ucsc.edu http://pacificrockyintertidal.org http://intertidalmap.org