OAI-ORE and BagIT development

[qqmyers]

This is an issue to track feedback related to developing a way to archive published datasets in DPN (http://dpn.org). I've done some proof-of-concept work to generate an OAI-ORE map file and BagIt bag (which uses and includes the ORE map file) for published datasets that I hope can form the basis for a DPN submission. From https://groups.google.com/forum/#!topic/dataverse-community/NZydpK_zXO0 :

I've posted some documentation which describes the use case and design rationale and has a run-down of some of the choices we've made to get to a proof-of-concept and some open issues. That documentation links to two example files - a json-ld ORE map and a BagIt bag for a test dataset. (FWIW: We're developing in the QDR fork of Dataverse at https://github.com/QualitativeDataRepository/dataverse/tree/feature/QDR-953

[qqmyers]

After discussions at the 2018 Dataverse meeting (thanks!), I've tried to identify a list of things to get to a minimum viable solution. Comments welcome.

ORE updates:

• add URI (and/or block-level namespace) to *.tsv import/api/metadata block model, use that URI, if it exists, instead of generated one in oremap (keeping a generated one as the default for tsv files that have not been updated)
• add sha256 (and 512) as options to Dataverse, create a way to generate for existing files (e.g. with check of existing md5)?
• decide on mapping of terms to URIs for internal DV metadata and citation.tsv metadata, with options being DDI(disco?), DCTerms, schema.org, and/or custom Dataverse terms (as the current code has). If there are custom terms, create a page at the URL used.
• submit OAI-ORE exporter to Dataverse (i.e. make it available as an option in the export menu) - this will probably include an update to allow exporters to stream instead of generating a string in memory.

Bag updates:

• Adapt bag to follow RDA guidance at https://www.rd-alliance.org/system/files/Research%20Data%20Repository%20Interoperability%20WG%20-%20Final%20Recommendations_0.pdf (move metadata files to a /metadata subdir, add a Datacite.xml file, include a tagmanifest file, change name of oremap file per spec, identify a minimal bagit profile to reference)?

DPN updates:

• Still TBD - planning discussion with them to decide whether to submit an ~RDA bag to be wrapped in a DPN bag, to submit a DPN bag directly, or perhaps to use the ORE map file to construct a bag via DPN's API.

[qqmyers]

An update: w.r.t. code - the URI and namespace for metadatablocks and support for sha256/512, including an api call to verify the existing hash and replace it, if the file is still valid, with one form the new algorithm, are both done/in the branch. I'm currently working on default URIs for the citation metadata block, and adapting the bag.

John Crabtree and I had a good call with Dave Pcolar of DPN today and it appears that either sending individual files or an ~RDA bag, either of which would be wrapped in a DPN bag for preservation, should be doable today. Sending a DPN bag directly is something that DPN is willing to work on, but is not currently supported. From the discussion, it appears that sending files directly could have higher performance (due to parallel transfer) but the idea of using an ~RDA bag as a general export, common intermediate/standard across possible preservation systems sounds compelling and I think sending a bag is currently the consensus option. We had some discussion of potential next steps w.r.t. versioning (perhaps just publishing the version changes given Dataverse's ability to identify them) and how to assure that variable level metadata is included (by including the ddi metadata file and/or adding to the ORE map).

[qqmyers]

After a second discussion with DPN and Odum, I've gone ahead with a consensus plan to enable optional submission to DPN as a post-publication Dataverse workflow as a v 1 effort. Based on how DPN works, including the fact that initial submission is synchronous and reversible while creating a 'snapshot' to archive a space can have a delay and is irreversible (except for manual removal), the workflow creates a space, named after the dataset globalId and uploads a BagIt bag, named after the globalId+ version, and a datacite.xml file to it. The success of this step is reported in a new column on the versions tab, visible only to admins, that reports failure or provides a link to find the data in the DPN Duracloud admin GUI. A curator would click the button to create a snapshot and monitor progress from there. Once the snapshot exists, the space is automatically emptied and can be deleted. Publishing a new version of a dataset will recreate the space and the process can be repeated with a snapshot of the new Bag and datacite.xml file. (Versions are therefore stored as different snapshots of the same space.)

I've been testing this in a 4.9.2-based QDR branch and it works reliably, though I did hit a DPN bug at one point. As a side-effect of the main effort, the datacite.xml file can be made available as a metadata export format (and it may be worth looking at it to add more fields as we just did with the citation formats). I've removed the Bag generation from the metadata export menu where I initially tested it for several reasons - it's not just metadata, it includes restricted files and access to it should be restricted, it's better to stream it to DPN/generate on demand rather than caching it (as its similar in size to the whole dataset).

I have a few things to finish up before this is ready for review/QA:

• suggesting vocabulary matches for terms in citation.tsv
• add a place for admins to download a Bag if/when the workflow fails
• add a way to archive already published datasets
• add the datacite.xml file to the bag and check for any other RDA recommended changes
• make the update available as a PR off the current develop branch

If anyone would like to see it early, I'd be happy to demo/discuss.

[pdurbin]

A curator would click the button to create a snapshot

@qqmyers what would label on the button be? If it's easy to provide a screenshot of what you have so far, can you please add it here? Today in a design meeting we were thinking about UI impact on the dataset page and I mentioned that at one point you were planning to put a button under "Export" even though we might want to consider a different place and name for it.

[qqmyers]

@pdurbin - the button you reference is one in the duracloud admin webapp, not Dataverse. I was originally thinking an 'export button' on the dataset page would be good, but since it could contain restricted files and is version specific, I've gone for the admin-only column in the version table, which is currently something like below. Non-admins would just see the normal table. All that said, it's all just 'proposed' and I'm definitely interested in feedback w.r.t. GUI, as well as who should be able to access the 'retry' functionality, etc. etc.

[pdurbin]

@qqmyers thanks, I've been talking to @mheppler a bit about your screenshot.

I know you wrote extensive documentation about what you're up in the "Data and Metadata Packaging for Archiving" that's ultimately linked from the Google Group post in the description of this issue but here's a direct link: https://github.com/QualitativeDataRepository/dataverse/wiki/Data-and-Metadata-Packaging-for-Archiving

People should also check out the discussion about the plan in the Google Group thread: https://groups.google.com/d/msg/dataverse-community/NZydpK_zXO0/vuvhnHL7AQAJ

Another resource is @qqmyers 's talk at the 2018 Dataverse Community Meeting. "Dataverse and Preservation in the Qualitative Data Repository" at https://drive.google.com/open?id=1fVhtw-R3Jf7wO4tgkNxpk3Mm93bUIjXP

[qqmyers]

@pdurbin - FYI - I've extended the workflow mechanism as discussed on the community call to allow system settings and an apiKey for the current user to be sent to a workflow step and, after some EJB fun, I think I have DPN submission as a workflow working along with the ability to submit past versions via the GUI. I have some cleanup to do, but I'm about ready to submit a PR(s) and would like to ask: There are a few things like the workflow changes and making the export mechanism use streaming data that were needed for DPN submission but could be submitted and reviewed as separate PRs. Would it be helpful to do that? That could be a little extra work for me, but I don't think it's that much since I have to compare between QDR's 4.9.2-based branch and develop anyway. It may help with review, but there would be dependencies between the PRs too. Let me know what you all think. Thanks!

[djbrooke]

@scolapasta any thoughts/guidance re: the approach in the above comment from @qqmyers?

hooray workflows!

[scolapasta]

@qqmyers yes, generally, having separate, smaller PRs are easier for us to review, QA, and merge. So since it isn't too much work on your side, we would prefer that approach.

[pdurbin]

@qqmyers I saw you made pull request #5049 and I assume it's the main one for this issue so I dragged it to code review at https://waffle.io/IQSS/dataverse

[djbrooke]

Hi @qqmyers - thanks for talking about this earlier this week. The other PRs are being reviewed. The workflow-based integration here will be extremely useful and is a fulfillment of a long-standing community need.

I have some concerns about the UI piece here. We’ll have a lot of moving pieces on the dataset and file pages as part of the redesign effort, so we don’t want to add any additional UI elements to the page right now, even if it’s only for superusers. It doesn’t appear there are API endpoints for the archiving via the UI that’s shown in the screenshot above. If these endpoints could be added, I think it would allow the desired functionality while not adding additional challenge to the design team's work in flight.

Let me know if you have any thoughts on the above. Thanks for all the PRs!

[qqmyers]

@djbrooke - there is an api, just missed it in the merge (and just added). For QDR, I think the ability for curators to see the status and be able to send things via the GUI will be important, but I can pull that part from the PR.

[djbrooke]

Thanks @qqmyers. I saw some commits come in on the associated PR over the weekend. Is this ready for code review? Let me know if you'd like us to take another look.

[qqmyers]

@djbrooke - I've added the api and removed the gui part, so this could be reviewed. However, because this PR depends on/includes the others, you should only review the parts not being covered elsewhere or wait until I can address the comments in other PRs and remerge everything into this one.

[qqmyers]

@djbrooke - With the other PRs merged, I think this should be ready for review. (FWIW: I made some minor updates- to have the API use the dataset DOI and friendly version number rather than the datasetversion id from the database, and changed from requiring superuser to using a new permission which is assigned only to admin by default. I also added documentation in the guides.)

[djbrooke]

Thanks @qqmyers we'll take a look!

[pdurbin]

@qqmyers one thing that caught my eye in pull request #5049 is that you've added an additional permission (ArchiveDatasetVersion). We haven't done this since we released 4.0 and it was my understanding that if/when we ever do, we would need to bump up the "permissionbits" for the "admin" role from 8191 to a higher number that includes the bits for the new permission. Otherwise, for installations that are upgrading, the "admin" role won't have the new permission. Maybe a screenshot of the "dataverserole" table will help:

[qqmyers]

@pdurbin - yep - there's an add to the update sql to go to 16383...

[pdurbin]

Wow, you've thought of everything. 😄

[qqmyers]

That remains to be seen :-) Glad you're checking things...

[kcondon]

@qqmyers Is this pr still viable? I ask because it appears that dpn is no longer available, right? Also, I was told your github repo for this contained some useful notes on use cases but it throws a 404. Last, is there a test archive that I can use to verify end to end workflow?

Thanks,

Kevin

[qqmyers]

@kcondon - absolutely. One of the three options DPN gives for going forward is to use the same duracloud interface to go to Chronopolis. So - no code changes here and I've already updated the docs to indicate that this is relevant to Duracloud/Chronopolis. I also expect that the BagIt code, which is in this PR, will be valuable as a way to package for other archives as well. The URL for notes at github is https://github.com/QualitativeDataRepository/dataverse/wiki/Data-and-Metadata-Packaging-for-Archiving, which works for me. If you can't get to it, we may have some permission issue, so let me know. I'm not sure about a test archive/account. I can look into it, but if not I could probably share the account I use with a temporary password (the automated workflow ends with the copy in Duracloud. It's a manual step to create a 'snapshot' which pushes a copy to permanent storage in Chronopolis. So, as long as we don't do the last manual step, test data can easily be deleted.)

[qqmyers]

also - per review discussion, I also made the archiver mechanism extensible, so it should be easier to add new archiver classes to send content to other long-term repositories going forward.

[kcondon]

@qqmyers OK, makes sense, thanks. I can get to your notes, I'll read them over. How would you recommend I test it if not with a test archive? I could do basic regression testing and try configuring it but does not seem to exercise the code, right? No mock or anything I suppose. I wouldn't need to move it to long term storage, just invoke the api and the workflow.

[qqmyers]

@kcondon - I think bouncing off a test archive is the only way to fully test. (You can see the oremap and datacite files from the export metadata menu, but there's no way to get the Bag except by sending to Duracloud). I'm checking on whether/how we can get you a temporary account to bounce off qdr.duracloud.org. It also looks like Duracloud offers a 12 hour free trial that should work (which I assume then gets you a sales call follow-up).

[qqmyers]

@kcondon - I guess it would be good to have you try the 12 hour trial from https://duraspace.org/duracloud/trial-account-request/ - if that works, others can do that to test/demo and I can make a note in the docs. If not, just ping me and I can create a temp account to access qdr.duracloud.org.

[pdurbin]

At standup I mentioned that I ran the following...

./ec2-create-instance.sh -b IQSS/4706-DPN_Submission_of_archival_copies -r https://github.com/QualitativeDataRepository/dataverse.git

... to spin up pull request #5049 and it seems to have worked. Here's the output:

To ssh into the new instance: ssh -i key-pdurbin-CE7E623C.pem centos@ec2-54-161-44-72.compute-1.amazonaws.com Branch "IQSS/4706-DPN_Submission_of_archival_copies" from https://github.com/QualitativeDataRepository/dataverse.git has been deployed to http://ec2-54-161-44-72.compute-1.amazonaws.com:8080 When you are done, please terminate your instance with: aws ec2 terminate-instances --instance-ids i-042da63d54d73ddcd

Here's a screenshot (note "v. 4.10.1 build IQSS/4706-DPN_Submission_of_archival_copies-b7b0a68" in the bottom right):

I'm happy to make the private key available to anyone who wants it. Well, not anyone but you know what I mean.

Today I learned that the default "dataverseAdmin" password is set by dataverse-ansible and is different that what we're used to. I'm happy to give that out as well.

[pdurbin]

There's been a lot of merging today so that EC2 instance I spun up is getting a bit out of date. I'll probably delete it soon. If anyone wants to try the ec2-create-instance.sh command above, please see http://guides.dataverse.org/en/4.10.1/developers/deployment.html#deploying-dataverse-to-amazon-web-services-aws

@qqmyers so far the only merge conflict is in the guides but do you mind merging the latest from develop?

[pdurbin]

@qqmyers just merged the latest from develop into his pull request: #5049

I guess I'll move this to code review? I've been talking to @djbrooke a bit on the side about the EC2 stuff above.

[kcondon]

@qqmyers I'm getting an error trying to load a thumbnail that's on s3. This works on /develop:

[2019-01-14T11:54:22.838-0500] [glassfish 4.1] [WARNING] [] [edu.harvard.iq.dataverse.api.ApiBlockingFilter] [tid: _ThreadID=55 _ThreadName=jk-connector(4)] [timeMillis: 1547484862838] [levelValue: 900] [[ Error processing /api/v1/datasets/:persistentId/thumbnail: org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.partitions.PartitionsLoader javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.partitions.PartitionsLoader at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:391) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:344) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:226) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at edu.harvard.iq.dataverse.api.ApiBlockingFilter.doFilter(ApiBlockingFilter.java:162) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at edu.harvard.iq.dataverse.api.ApiRouter.doFilter(ApiRouter.java:30) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:873) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:739) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:575) at org.apache.catalina.core.ApplicationDispatcher.doDispatch(ApplicationDispatcher.java:546) at org.apache.catalina.core.ApplicationDispatcher.dispatch(ApplicationDispatcher.java:428) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:378) at edu.harvard.iq.dataverse.api.ApiRouter.doFilter(ApiRouter.java:34) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:316) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:412) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282) at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167) at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201) at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175) at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235) at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561) at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545) at java.lang.Thread.run(Thread.java:748) Caused by: org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.partitions.PartitionsLoader at org.glassfish.jersey.servlet.internal.ResponseWriter.rethrow(ResponseWriter.java:256) at org.glassfish.jersey.servlet.internal.ResponseWriter.failure(ResponseWriter.java:238) at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:439) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) at org.glassfish.jersey.internal.Errors.process(Errors.java:315) at org.glassfish.jersey.internal.Errors.process(Errors.java:297) at org.glassfish.jersey.internal.Errors.process(Errors.java:267) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1028) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:372) ... 53 more

[qqmyers]

@kcondon - another odd one. PartitionsLoader is in the aws-java-sdk-core-1.11.172.jar file which, when I compile this branch, is in WEB-INF/lib as it should be. And I don't see the issue in my working branches. Do you see this jar in your deployment? I see it and 2 other aws-sdk-* files with the same version number. Scanning through all of the other jars, the only difference I saw was the version of commons-compress, which I had updated locally. I just committed that change here in case it is somehow related...

[kcondon]

@qqmyers I've rebuilt but see the same problem. As for the libs, here is what I'm seeing:

pwd /usr/local/glassfish4/glassfish/domains/domain1/applications/dataverse-4.10.1/WEB-INF/lib

ls -ltr aws* -rw-r--r--. 1 root root 67129833 Jan 14 13:36 aws-java-sdk-bundle-1.11.172.jar -rw-r--r--. 1 root root 848498 Jan 14 13:36 aws-java-sdk-core-1.11.172.jar -rw-r--r--. 1 root root 279962 Jan 14 13:36 aws-java-sdk-sqs-1.11.172.jar

[qqmyers]

@kcondon - the middle 'core' jar is the one in which I see the com.amazonaws.partitions.PartitionsLoader class and your file size of 848498 is the same that I see. Could there be permission issues? I'm not sure what else could cause glassfish to not be able to read through a jar that is where it should be...

[kcondon]

@qqmyers I'm not sure but I don't think so since a build from /develop on the same box works. I'll compare files from both builds to be sure.

[kcondon]

@qqmyers When I deploy /develop I only see the one file: ls -ltr aws* -rw-r--r--. 1 root root 67129833 Jan 14 13:52 aws-java-sdk-bundle-1.11.172.jar

[qqmyers]

@kcondon - a good clue. While I can run with all 3 jars there, I see that the class in question is also in the sdk-bundle, so that duplication must be an issue. In any case, I tracked down where the other aws jars are coming from (dependencies within the duracloud storeclient) and removed them in the latest commit. Hopefully that works for you.

[kcondon]

@qqmyers That fixed it, thanks. Btw, I should be able to run this branch with only the db update if I choose not to configure archiving, correct? Also, the db update script name is old/ out of date and only the one line for this branch needs to be preserved.

[qqmyers]

@kcondon - I think so. In fact, I think the db update doesn't have to be done unless you set this up - the archivalcopylocation only gets set if you archive and, without the GUI mods to the version table that were dropped from the PR, it should never be read either. And by db update script out-of-date - you mean that this line should pop out of upgrade_v4.9.4_to_4.10.0.sql and into whatever v4.10.1 to ? comes next? If so, I agree - is there a naming convention for that next file (just assume a v4.10.2 for now?). I can move it if so.

[kcondon]

@qqmyers OK, good to know. I had seen exceptions when running reexportall endpoint and thought it was due to missing db column but will retest. As for naming next release, we're calling it 4.11 as far as github milestones are concerned. So, name it upgrade_v4.10_to_v4.11.sql

[kcondon]

@qqmyers OK, got my Duracloud test account and configured the settings and jvm-options as best I could. I've left port and context as defaults.

When I run the api on an existing dataset, I get the html for an http 500 page. The server.log says,

[2019-01-14T16:26:32.576-0500] [glassfish 4.1] [WARNING] [AS-EJB-00056] [javax.enterprise.ejb.container] [tid: _ThreadID=31 _ThreadName=http-listener-1(4)] [timeMillis: 1547501192576] [levelValue: 900] [[ A system exception occurred during an invocation on EJB EjbDataverseEngine, method: public java.lang.Object edu.harvard.iq.dataverse.EjbDataverseEngine.submit(edu.harvard.iq.dataverse.engine.command.Command) throws edu.harvard.iq.dataverse.engine.command.exception.CommandException]]

[2019-01-14T16:26:32.576-0500] [glassfish 4.1] [WARNING] [] [javax.enterprise.ejb.container] [tid: _ThreadID=31 _ThreadName=http-listener-1(4)] [timeMillis: 1547501192576] [levelValue: 900] [[

javax.ejb.TransactionRolledbackLocalException: Exception thrown from bean at com.sun.ejb.containers.EJBContainerTransactionManager.checkExceptionClientTx(EJBContainerTransactionManager.java:662) at com.sun.ejb.containers.EJBContainerTransactionManager.postInvokeTx(EJBContainerTransactionManager.java:507) at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4566) at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2074) at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2044) at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220) at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88) at com.sun.proxy.$Proxy317.submit(Unknown Source) at edu.harvard.iq.dataverse.EJB31_GeneratedEjbDataverseEngineIntf__Bean.submit(Unknown Source) at edu.harvard.iq.dataverse.api.Admin.submitDatasetVersionToArchive(Admin.java:1321) at sun.reflect.GeneratedMethodAccessor409.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081) at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153) at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4786) at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64) at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52) at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140) at sun.reflect.GeneratedMethodAccessor125.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822) at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:369) at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4758) at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4746) at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212) at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88) at com.sun.proxy.$Proxy303.submitDatasetVersionToArchive(Unknown Source) at edu.harvard.iq.dataverse.api.EJB31_GeneratedAdminIntf__Bean__.submitDatasetVersionToArchive(Unknown Source) at sun.reflect.GeneratedMethodAccessor409.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:152) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:387) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:331) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:103) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) at org.glassfish.jersey.internal.Errors.process(Errors.java:315) at org.glassfish.jersey.internal.Errors.process(Errors.java:297) at org.glassfish.jersey.internal.Errors.process(Errors.java:267) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1028) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:372) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:344) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:226) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at edu.harvard.iq.dataverse.api.ApiBlockingFilter$3.doBlock(ApiBlockingFilter.java:65) at edu.harvard.iq.dataverse.api.ApiBlockingFilter.doFilter(ApiBlockingFilter.java:157) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at edu.harvard.iq.dataverse.api.ApiRouter.doFilter(ApiRouter.java:30) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:873) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:739) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:575) at org.apache.catalina.core.ApplicationDispatcher.doDispatch(ApplicationDispatcher.java:546) at org.apache.catalina.core.ApplicationDispatcher.dispatch(ApplicationDispatcher.java:428) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:378) at edu.harvard.iq.dataverse.api.ApiRouter.doFilter(ApiRouter.java:34) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:316) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:412) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282) at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167) at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201) at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175) at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235) at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561) at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException at edu.harvard.iq.dataverse.engine.command.impl.AbstractSubmitToArchiveCommand.execute(AbstractSubmitToArchiveCommand.java:37) at edu.harvard.iq.dataverse.engine.command.impl.AbstractSubmitToArchiveCommand.execute(AbstractSubmitToArchiveCommand.java:21) at edu.harvard.iq.dataverse.EjbDataverseEngine.submit(EjbDataverseEngine.java:232) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081) at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153) at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4786) at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64) at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52) at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140) at sun.reflect.GeneratedMethodAccessor125.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822) at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:369) at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4758) at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212) ... 110 more ]]

[qqmyers]

@kcondon (AbstractSubmitToArchiveCommand.java:37) is where it looks for the archiver settings. Did you do: curl http://localhost:8080/api/admin/settings/:ArchiverSettings -X PUT -d ”:DuraCloudHost, :DuraCloudPort, :DuraCloudContext” ?

[kcondon]

@qqmyers OK, that was missing. I've called it as is in the doc but was a little confused by the statement: This setting is a comma-separated list of those settings. Note that this list must include the :ArchiverClassName setting.

I did not see ArchverClassName in the example command line for :ArchiverSettings

Also, the example includes 3 other settings it says must also be created but then says 2 may be default. So, if they are default do I add them to ArchiverSettings or leave them out?

It throws a new stack trace. Have I just got the port/context wrong?

[2019-01-14T16:48:19.144-0500] [glassfish 4.1] [WARNING] [] [edu.harvard.iq.dataverse.engine.command.impl.DuraCloudSubmitToArchiveCommand] [tid: _ThreadID=30 _ThreadName=http-listener-1(3)] [timeMillis: 1547502499144] [levelValue: 900] [[ Error attempting to create space 'doi-10-5072-fk2-sc05oe' due to: User is not authorized to perform the requested function]]

[2019-01-14T16:48:19.149-0500] [glassfish 4.1] [SEVERE] [] [] [tid: _ThreadID=30 _ThreadName=Thread-9] [timeMillis: 1547502499149] [levelValue: 1000] [[ org.duracloud.error.UnauthorizedException: Error attempting to create space 'doi-10-5072-fk2-sc05oe' due to: User is not authorized to perform the requested function at org.duracloud.client.ContentStoreImpl.doCreateSpace(ContentStoreImpl.java:404) at org.duracloud.client.ContentStoreImpl.access$300(ContentStoreImpl.java:68) at org.duracloud.client.ContentStoreImpl$5.retry(ContentStoreImpl.java:387) at org.duracloud.client.ContentStoreImpl$5.retry(ContentStoreImpl.java:383) at org.duracloud.common.retry.Retrier.execute(Retrier.java:135) at org.duracloud.client.ContentStoreImpl.execute(ContentStoreImpl.java:239) at org.duracloud.client.ContentStoreImpl.createSpace(ContentStoreImpl.java:383) at edu.harvard.iq.dataverse.engine.command.impl.DuraCloudSubmitToArchiveCommand.performArchiveSubmission(DuraCloudSubmitToArchiveCommand.java:77) at edu.harvard.iq.dataverse.engine.command.impl.AbstractSubmitToArchiveCommand.execute(AbstractSubmitToArchiveCommand.java:52) at edu.harvard.iq.dataverse.engine.command.impl.AbstractSubmitToArchiveCommand.execute(AbstractSubmitToArchiveCommand.java:21) at edu.harvard.iq.dataverse.EjbDataverseEngine.submit(EjbDataverseEngine.java:232) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081) at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153) at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4786) at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64) at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52) at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140) at sun.reflect.GeneratedMethodAccessor125.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822) at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:369) at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4758) at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4746) at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212) at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88) at com.sun.proxy.$Proxy317.submit(Unknown Source) at edu.harvard.iq.dataverse.EJB31_GeneratedEjbDataverseEngineIntf__Bean.submit(Unknown Source) at edu.harvard.iq.dataverse.api.Admin.submitDatasetVersionToArchive(Admin.java:1321) at sun.reflect.GeneratedMethodAccessor409.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081) at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153) at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4786) at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:64) at org.jboss.weld.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52) at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140) at sun.reflect.GeneratedMethodAccessor125.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:822) at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:369) at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4758) at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4746) at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212) at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88) at com.sun.proxy.$Proxy303.submitDatasetVersionToArchive(Unknown Source) at edu.harvard.iq.dataverse.api.EJB31_GeneratedAdminIntf__Bean.submitDatasetVersionToArchive(Unknown Source) at sun.reflect.GeneratedMethodAccessor409.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:152) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:387) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:331) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:103) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) at org.glassfish.jersey.internal.Errors$]]

[2019-01-14T16:48:19.149-0500] [glassfish 4.1] [SEVERE] [] [] [tid: _ThreadID=30 _ThreadName=Thread-9] [timeMillis: 1547502499149] [levelValue: 1000] [[ 1.call(Errors.java:267) at org.glassfish.jersey.internal.Errors.process(Errors.java:315) at org.glassfish.jersey.internal.Errors.process(Errors.java:297) at org.glassfish.jersey.internal.Errors.process(Errors.java:267) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1028) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:372) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:344) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:226) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at edu.harvard.iq.dataverse.api.ApiBlockingFilter$3.doBlock(ApiBlockingFilter.java:65) at edu.harvard.iq.dataverse.api.ApiBlockingFilter.doFilter(ApiBlockingFilter.java:157) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at edu.harvard.iq.dataverse.api.ApiRouter.doFilter(ApiRouter.java:30) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:873) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:739) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:575) at org.apache.catalina.core.ApplicationDispatcher.doDispatch(ApplicationDispatcher.java:546) at org.apache.catalina.core.ApplicationDispatcher.dispatch(ApplicationDispatcher.java:428) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:378) at edu.harvard.iq.dataverse.api.ApiRouter.doFilter(ApiRouter.java:34) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:316) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:412) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282) at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167) at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201) at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175) at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235) at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561) at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545) at java.lang.Thread.run(Thread.java:748) Caused by: org.duracloud.error.UnauthorizedException: User is not authorized to perform the requested function at org.duracloud.client.ContentStoreImpl.checkResponse(ContentStoreImpl.java:993) at org.duracloud.client.ContentStoreImpl.doCreateSpace(ContentStoreImpl.java:400) ... 149 more]]

[kcondon]

For the record I can directly log in to my Duracloud test account but it seems like I'm configuring the credentials Dataverse-side incorrectly.

[pdurbin]

@kcondon in 7b7f4b5ff I removed the smart quotes in the curl commands and @qqmyers and I talked about possibly reorganizing stuff in the future, perhaps in a follow up pull request. Jim's thoughts from IRC, which sounds good to me:

"I'm open as to where stuff goes, just need guidance or for you guys to move it as you see fit. The tricky part here is that the archiver mechanism is generic/extensible now and could be used by others. I could see the Duracloud part going to config, but the archiver mechanism is more developer-level?"

[kcondon]

@qqmyers I'm not 100% sure this test account can create spaces. So that might be the issue.

[kcondon]

@qqmyers Just heard back from Duracloud. Test account is very limited so likely cannot use since archiving creates a space with the doi in the name, correct? This was their response. I may be able to get them to help a bit more but so far dead in the water.

Hi Kevin,

The trial account capability is quite limited, and it has no admin features which are required for creating a space. If you'd like I can crate a space or spaces for you to use on our internal testing DuraCloud account. If you'll just let me know what you'd like to call the space or spaces, and make a profile and let me know the username, I am happy to set that up. You can make a profile here: https://manage.duracloud.org/users/new

[qqmyers]

@kcondon - the code is going to try to dynamically create a space and will fail if it already exists, so having it set up manually won't help. I just went ahead and added you to the qdr.duracloud account for testing (via email). Let me know if that doesn't work.

[pdurbin]

@qqmyers any thoughts on any doc re-org? It might be nice to move the details out of the integrations page and into the config page, similar to the Compute button.

[kcondon]

@qqmyers Thanks, was able to create an account but still getting 401s. I'll try their api to be sure. Fyi, I'm actually out sick today but wanted to at least try new creds to see whether we can get past this block.

When I log in to Duracloud I get:

DuraCloud Vault: Qualitative Data Repository	Status: INACTIVE

No instance associated with this account.This status of this account is currently INACTIVE. Please contact your administrator to make your account ACTIVE.

[qqmyers]

@kcondon - thanks for sticking with it. When I look at the Duracloud management console, I do see you listed as an active user now (not an admin if that's required to create spaces). One test would be to see if you can login at https://qdr.duracloud.org and see an 'add space' button that you can successfully create a space with. If not, I can try to add privileges for you. If that does work, I can send you the specific api call coming from the new code so you can try it with curl...