Add a NMFS BagIt option to ArchiveADataset

[BobSimons]

ERDDAP comes with a tool called ArchiveADataset which extracts a subset of a dataset (perhaps in multiple files) and packages the results in a zip or tgz file in a format suitable for submission to NOAA's NCEI archive (or for other purposes). Unfortunately, NCEI hasn't standardized the format they want. So different groups inside and outside of NCEI have requested different formats and I have added support for them to ArchiveADataset as they have arisen. This latest request is a collaboration between NOAA's NMFS and NCEI. It is very similar to the existing BagIt option in ArchiveADataset, but they want to be able to include other files in the BagIt file.

If you chose to work on this, you'll contact and work with Nazila Merati of NMFS to make sure you make the format that they want.

Skills required: Java programming.

Difficulty: Relatively easy. ArchiveADataset already exists and has several output options. You'll just make a clone of one of them that works a little differently. Perhaps 2 weeks work (after you figure out ERDDAP and ArchiveADataset).

Mentor: Bob Simons (main author of ERDDAP)

Please also read the Programmer's Guide at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#programmersGuide especially the "Judging Your Code Contributions" section.

[jarvis-001]

Hi @BobSimons I have read a lot about ERDDAP and technical stuff related to oceanographic data like various data formats used to store data like HDF5(commonly used and oldest) , Zarr (multi processing possible means faster data retrieval and processing ), n5 etc, OPeNDAP ( framework that simplifies all aspects of scientific data networking and makes local data accessible to remote locations ) etc. I also read the resources you gave along with "Judging Your Code Contributions" section and am highly interested to contribute to the project . So are there any tasks I need to do to contribute to the project ?

[BobSimons]

Excellent! Thanks for your interest in doing this.

1) If you want to do this as part of Google Summer of Code (GSoC), then we'll follow a slightly different route versus if you want to just do it, now, on your own. Please let me know if you want to do this as part of GSoC.

2) In either case, please

• Install ERDDAP on your PC. It needs to have at least one active dataset (in addition to the special "AllDatasets")
• Please read about ArchiveADataset https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#ArchiveADataset
• Run ArchiveADataset a few times so you understand what it does and how to use it.
• Then email me to tell me you got this far.
• If you are doing this for GSoC, you will then write up your GSoC application. If not, you'll just start the project.
• For the actual project, we'll contact the person at NOAA NMFS who requested this feature to get the specifics of what she wants. Note that most of the hard part (e.g., creating ArchiveADataset and making tools to make archives) is done. You just need to add a new option for creating a new type of archive by using the underlying tools in a slightly different way.

[jarvis-001]

@BobSimons I did try multiple times downloading ERDDAP file but I think it is hosted on physical servers because of which connection is constantly breaking . What I believe is that the server on which erddap.war is hosted is using a single connection instead of chunks . Even when download usind download manager while downloading it is forming single connection rather than downloding in chunks due to which even when for a second if the connection is lost, entire downloaded data is lost and when the connection is re-established, the whole file starts downloading again .

[BobSimons]

I assume you are referring to the erddap.war download from https://github.com/BobSimons/erddap/releases/download/v2.11/erddap.war Yes, it is a big file.

I switched to distributing it via GitHub because that was faster and more reliable. I don't have an alternative.

Can you go to a place with more a reliable internet connection, e.g., a university?

[jarvis-001]

Ohh thanks @BobSimons now it's done and am working on setting erddap in my system

[jarvis-001]

Hi @BobSimons am getting an issue installing it with windows am not able to run chown command on windows and commands on windows and on linux I can't access the file So what should I do now ? Like now I am installing wsl on my windows but it will take some time

[BobSimons]

Please see my comments below...

On Mon, Apr 12, 2021 at 1:07 AM jarvis-001 @.***> wrote:

Hi Bob am getting an issue installing it with windows am not able to run chown command on windows and commands on windows and on linux I can't access the file

I didn't know which operating system you had. chown is for Linux. For windows, permissions problems are usually solved by right clicking on files and working with menus. I don't know the specific problem so I can give a specific solution.

Do you have Linux or windows? or both on different computers?

In general, search Google for solving general errors, e.g. permissions problems.

So what should I do now ?

Like now I am installing wsl on my windows but it will take some time

If you have Windows, do everything with Windows. You don't need Linux or WSL. My PC is Windows.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/BobSimons/erddap/issues/45#issuecomment-817587590, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALKWOCZ4JGW5ZWOJ435OYDTIKS3DANCNFSM4XBNLU3Q .

[jarvis-001]

Ohh ok Bob I'll try again

[jarvis-001]

Hi Bob I have followed all the steps and ERDDAP is around set in my device . So may I write my proposal accordingly ? And since this issue is a smaller one I was thinking of working on any other issue too which is similar to this one. And I will set the environment on my device and all the pre requisites before the community bonding period starts (maybe by 24 April ) .

[jarvis-001]

Basically I was thinking of contributing to issue #48 along with this .

[BobSimons]

Yes, Please write the GSoC proposal. I think the deadline is today (April 13). Yes, adding another small issue in case you have time is a good idea. #48 is appropriate.

[jarvis-001]

Hi Bob Thank you so much for your constant support and help during the entire time . It made thongs a lot easier to understand ...

[jarvis-001]

Hi Bob, Actually I will be having my end term examinations during community bonding period. So is it possible that we do that work before then? If not possible than no problem

On Tue, 13 Apr 2021, 19:57 Bob Simons, @.***> wrote:

Yes, Please write the GSoC proposal. I think the deadline is today (April 13). Yes, adding another small issue in case you have time is a good idea. #48 https://github.com/BobSimons/erddap/issues/48 is appropriate.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/BobSimons/erddap/issues/45#issuecomment-818780396, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOX6QDZGUVQPZ5QYZF4YR4DTIRIFXANCNFSM4XBNLU3Q .

[BobSimons]

I will keep that in mind. We (IOOS) still have to select the projects that will proceed. We don't yet know how many slots/internships we get.

On Fri, Apr 16, 2021 at 2:59 PM jarvis-001 @.***> wrote:

Hi Bob, Actually I will be having my end term examinations during community bonding period. So is it possible that we do that work before then? If not possible than no problem

On Tue, 13 Apr 2021, 19:57 Bob Simons, @.***> wrote:

Yes, Please write the GSoC proposal. I think the deadline is today (April 13). Yes, adding another small issue in case you have time is a good idea. #48 https://github.com/BobSimons/erddap/issues/48 is appropriate.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/BobSimons/erddap/issues/45#issuecomment-818780396, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AOX6QDZGUVQPZ5QYZF4YR4DTIRIFXANCNFSM4XBNLU3Q

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/BobSimons/erddap/issues/45#issuecomment-821585096, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALKWOAYKBLKCWXJF3K5MT3TJCXLLANCNFSM4XBNLU3Q .

[jarvis-001]

Ohh sure Bob

On Sat, 17 Apr 2021, 04:34 Bob Simons, @.***> wrote:

I will keep that in mind. We (IOOS) still have to select the projects that will proceed. We don't yet know how many slots/internships we get.

On Fri, Apr 16, 2021 at 2:59 PM jarvis-001 @.***> wrote:

Hi Bob, Actually I will be having my end term examinations during community bonding period. So is it possible that we do that work before then? If not possible than no problem

On Tue, 13 Apr 2021, 19:57 Bob Simons, @.***> wrote:

Yes, Please write the GSoC proposal. I think the deadline is today (April 13). Yes, adding another small issue in case you have time is a good idea.

48

https://github.com/BobSimons/erddap/issues/48 is appropriate.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/BobSimons/erddap/issues/45#issuecomment-818780396 , or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AOX6QDZGUVQPZ5QYZF4YR4DTIRIFXANCNFSM4XBNLU3Q

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/BobSimons/erddap/issues/45#issuecomment-821585096, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AALKWOAYKBLKCWXJF3K5MT3TJCXLLANCNFSM4XBNLU3Q

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/BobSimons/erddap/issues/45#issuecomment-821708135, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOX6QD43TOBNTF5UF27QP7LTJC67VANCNFSM4XBNLU3Q .