As an API client user, I want to consistently and robustly start local servers for development and testing

[nutjob4life]

While developing for the PDS Deep Archive I needed to run an instance of the PDS API server locally. However, going about getting this running proved to be no walk in the park. The structure of the code, container deployment, and the various README files sure could use some polish.

• It looks like there are a number of "building block" repositories that—when put together—run https://pds-gamma.jpl.nasa.gov/api/ or your own pds-gamma-like registry API. But it's not clear how they relate. Is there a diagram? An explanation? Documentation?
• https://github.com/NASA-PDS/pds-api seems like it'd be the starting point, but README.md there has broken links (404 via app.swaggerhub.com, ithub.com without the g) and not much else. The links should be fixed and the overview of what-comprises-what should be here.
• https://github.com/NASA-PDS/pds-api describes "various implementation flavour" but it's confusing:
  • The four bullet items suggest there are "libraries" and "stub services" in Python and Java (2×2=4), but…
    • The "pds-api-javalib" is the "java library" but the "pds-api-pythonlib" is not the "python library" but the python stub service
      • (The "pds-api-pythonlib" has no code in it anyway)
  • There's instead a Python client—but no Java client? 🤷‍♀️
• https://github.com/NASA-PDS/pds-api-javalib is actually "PDS-API-BASE" 🤔
• https://github.com/NASA-PDS/registry-api-service is actually "PDS-API-SERVICE" 🧐
• https://github.com/NASA-PDS/pds-registry-app has an "Install" section that is "to be published"; it should be published.
• https://github.com/NASA-PDS/pds-registry-app has a "Docker" section that seems to work if you add -updateSchema n to the "ingest the harvested data" step. This should be documented in the README.
• https://github.com/nasa-pds/registry-api-service mentions the file src/main/resources/applications.properties but it's src/main/resources/application.properties.
  • It also should fill in the "Admimistrator" section in the documentation—and spell "Administrator" correctly 😅
• https://github.com/nasa-pds/registry-api-service suggests running mvn clean install sprint-boot:run but this results in a stack trace and perplexing error
  • The same documents give a Docker-flavored solution, but this fails with even longer stack traces
  • Changing the application.properties setting for elasticSearch.host to es:9200 and adding the --network pds argument to docker container run helps
• https://github.com/NASA-PDS/pds-api should have a docker-compose.yaml file that brings all this together
  • Running mvn inside a Dockerfile is just plain wrong. Docker images should only have those items needed to run the application, not build them.
  • hub.docker.com already has elasticsearch:7.10.1 so there's no reason it shouldn't have registry-api-service:1.2.3 as well
  • You can then remove the manual docker network create steps because the Docker Composition will take care of that!

[jordanpadams]

@nutjob4life unfortunately, the person to fix all of this is probably @tloubrieu-jpl (unless you figured it out all out 😎 ), but he is still out for several more weeks. can we maybe use the breakout tomorrow to walk through the issues you are seeing with @tdddblog and @jimmie and see what we can come up with / work real time?

I followed the docker deployment in pds-registry-app, which seemed to install "stuff", but I kept getting distracted by other things prior to testing. Additionally, I know for sure that if you used the latest stable release of any of the components, you will be unable to test pds-deep-archive, since those fixes were just merged to main.

that all being said, this is what I would really want in the end:

https://github.com/NASA-PDS/pds-api should have a docker-compose.yaml file that brings all this together

either in the pds-api or in the registry-app, somewhere that brings this all together would be ideal and I feel like it would make "bootstrapping" all these components into a pipeline dramatically easier.

now if we could just find more time and people to do this 🤔 ...

[jordanpadams]

as an additional note, were you able to find all the docs here? https://nasa-pds.github.io/pds-registry-app/ . That actually contains all the applicable documentation for installation, but I think you would just need to install the SNAPSHOT versions.

[tloubrieu-jpl]

Hi @nutjob4life , @jordanpadams is right the stating point is https://nasa-pds.github.io/pds-registry-app/ Since the API is 'only' the outgoing interface for the registry.

I will try to answer your points one by one:

• [x] It looks like there are a number of "building block" repositories that—when put together—run https://pds-gamma.jpl.nasa.gov/api/ or your own pds-gamma-like registry API. But it's not clear how they relate. Is there a diagram? An explanation? Documentation?

The best diagram related to your question can be found at https://nasa-pds.github.io/pds-registry-app/install/index.html

• [x] https://github.com/NASA-PDS/pds-api seems like it'd be the starting point, but README.md there has broken links (404 via app.swaggerhub.com, ithub.com without the g) and not much else. The links should be fixed and the overview of what-comprises-what should be here.

pds-api is about the PDS API standard. That is why it is not related to an implementation.

• [x] https://github.com/NASA-PDS/pds-api describes "various implementation flavour" but it's confusing:
• [ ] The four bullet items suggest there are "libraries" and "stub services" in Python and Java (2×2=4), but…
• [ ] The "pds-api-javalib" is the "java library" but the "pds-api-pythonlib" is not the "python library" but the python stub service
• [ ] (The "pds-api-pythonlib" has no code in it anyway)
• [x] There's instead a Python client—but no Java client? 🤷‍♀️

yes, no java client so far.

• [x] https://github.com/NASA-PDS/pds-api-javalib is actually "PDS-API-BASE" 🤔

I changed that

• [x] https://github.com/NASA-PDS/registry-api-service is actually "PDS-API-SERVICE" 🧐

I changed that

• [x] https://github.com/NASA-PDS/pds-registry-app has an "Install" section that is "to be published"; it should be published.

It is published so I added the link there

• [x] https://github.com/NASA-PDS/pds-registry-app has a "Docker" section that seems to work if you add -updateSchema n to the "ingest the harvested data" step. This should be documented in the README.

I updated the README.

• [x] https://github.com/nasa-pds/registry-api-service mentions the file src/main/resources/applications.properties but it's src/main/resources/application.properties.

thanks, I changed that.

• [x] It also should fill in the "Admimistrator" section in the documentation—and spell "Administrator" correctly 😅

I corrected the typo in adimnistrator, removed the deployment subsection which was the only one in dministrator section.

• [ ] https://github.com/nasa-pds/registry-api-service suggests running mvn clean install sprint-boot:run but this results in a stack trace and perplexing error

Could you share the stack trace ?

• [ ] The same documents give a Docker-flavored solution, but this fails with even longer stack traces

Same

• [x] Changing the application.properties setting for elasticSearch.host to es:9200 and adding the --network pds argument to docker container run helps

If you followed the instructions there https://github.com/NASA-PDS/registry-api-service/blob/main/README.md#docker

I added a prerequisite saying how to deploy the rest of the registry.

I added how to update the application.properties as you did. It can still be improved the documentation is still incomplete.

The --network pds option was lready there (that is why I am not sure if the documentation I looked at was the one you were refering to).

• [x] https://github.com/NASA-PDS/pds-api should have a docker-compose.yaml file that brings all this together

Actually I would have the docker-compose in https://github.com/NASA-PDS/pds-registry-app/ but I fully agree with that. I created a ticket for this task https://github.com/NASA-PDS/pds-registry-app/issues/186

We made this attempt on the tracking service and that drastically simplfy the documentation ! See https://github.com/NASA-PDS/tracking-service#docker

• [x] Running mvn inside a Dockerfile is just plain wrong. Docker images should only have those items needed to run the application, not build them.

ok, tht could be updated by downloading the released package, unzip and start java.

• [x] hub.docker.com already has elasticsearch:7.10.1 so there's no reason it shouldn't have registry-api-service:1.2.3 as well

Very good point, so we should make the docker image generation part of the continuous integration. I am 100% for that. I created this ticket https://github.com/NASA-PDS/pds-registry-app/issues/187

• [x] You can then remove the manual docker network create steps because the Docker Composition will take care of that!

ok, let's see that as part of the development of the docker-compose configuration.

Generally speaking I am feeling like, apart from your feedback on unaccurate README, the actions needed should be part of our organization of the docker configuration management as a whole as seen by Jimmy for the cloud depoyment. We should also integrate the work done by @tdddblog on the continuous integration of the registry in https://github.com/NASA-PDS/registry-ci. All of this should be centered in https://github.com/NASA-PDS/pds-registry-app which is the registry application including its web interface, the PDS API.

[nutjob4life]

Good morning and thanks to you two for the great read-through!

First, yes, I did figure it out and got a local working API. It helped me make progress on my feature and also discover a regression.

And I think that also qualifies me to help out with the points I mentioned! 😎 No need to spend time at the breakout meeting to get me up to speed: I assure you, I am 🏃‍♀️.

now if we could just find more time and people to do this 🤔 ...

Definitely sign me up for making a docker-compose.yaml for this as well as seeing if I can tweak the docs too.

Second, https://nasa-pds.github.io/pds-registry-app/ is a lovely resource for sure. Yet a lot of our websites lack visibility. As a developer, to me the "home page" for a repository is not the github.io page but the github.com/repo page, and I freely admit it never occurred to me to even look for such "end user" documentation. That's on me.

Maybe we can make it a part of our templates to have our README.mds look like this:

# 🪐 PDS Component XYZ

This is the XYZ that does this, that, and the other thing for the Planetary Data System.

Please visit our website at: https://nasa-pds.github.io/pds-xyz
It has useful information for developers and end-users.

## 📀 Installation

…

I know github.com automatically includes a link to the right, but I was totally blind to it. Sorry 😔

As for the diagram at https://nasa-pds.github.io/pds-registry-app/images/registry-harvest.png … yes that is a cool image. I love the anime character. But the diagram I'm after is a component diagram of the software, not a deployment diagram. How do pds-api, pds-api-javalib, pds-registry-app, registry-api-service, etc., all fit together. Unless I'm mistaken (which is in hindsight probably the case), there isn't a 1–1 correspondence.

pds-api is about the PDS API standard. That is why it is not related to an implementation.

Okay, let's make that clear in the README. I clearly was nosing around in the wrong area! I won't be the first person to make that mistake.

Could you share the stack trace ?

It's linked in the issue above; click/tap the link text "results in a stack trace and perplexing error". Same goes with the link text "even longer stack traces".

The --network pds option was lready there (that is why I am not sure if the documentation I looked at was the one you were refering to).

Maybe I just missed it. Please accept my apologies 🙇‍♀️

One thing that I'm still unclear on is: is it the "PDS API" or the "Registry API"? To me, Registry is just one function of the Planetary Data System, and the PDS may have a server with "validate" capabilities, "ingest" capabilities, etc. Is that coming in the future?

Thank you @tloubrieu-jpl for your thorough examination of the issue especially while on holiday. Please do not respond to this until you are back! You need a break from this! And thank you @jordanpadams for also leaping to this so quickly. It's obvious I had a lot of confusion and some amount of frustration but I got it working yesterday and some tweaks to the docs might make the next person's go-through smoother. And I'm happy to help with that.

[jordanpadams]

Good morning and thanks to you two for the great read-through!

First, yes, I did figure it out and got a local working API. It helped me make progress on my feature and also discover a regression.

And I think that also qualifies me to help out with the points I mentioned! 😎 No need to spend time at the breakout meeting to get me up to speed: I assure you, I am 🏃‍♀️.

+1 👍

now if we could just find more time and people to do this 🤔 ...

Definitely sign me up for making a docker-compose.yaml for this as well as seeing if I can tweak the docs too.

👍 assigned. after pds-deep-archive, we can think about where this fits into our priorities heading into the build delivery next week.

Second, https://nasa-pds.github.io/pds-registry-app/ is a lovely resource for sure. Yet a lot of our websites lack visibility. As a developer, to me the "home page" for a repository is not the github.io page but the github.com/repo page, and I freely admit it never occurred to me to even look for such "end user" documentation. That's on me.

Maybe we can make it a part of our templates to have our README.mds look like this:

# 🪐 PDS Component XYZ

This is the XYZ that does this, that, and the other thing for the Planetary Data System.

Please visit our website at: https://nasa-pds.github.io/pds-xyz
It has useful information for developers and end-users.

## 📀 Installation

…

👍 totally agree with this. just updated the template repos

I know github.com automatically includes a link to the right, but I was totally blind to it. Sorry 😔

As for the diagram at https://nasa-pds.github.io/pds-registry-app/images/registry-harvest.png … yes that is a cool image. I love the anime character. But the diagram I'm after is a component diagram of the software, not a deployment diagram. How do pds-api, pds-api-javalib, pds-registry-app, registry-api-service, etc., all fit together. Unless I'm mistaken (which is in hindsight probably the case), there isn't a 1–1 correspondence.

@tloubrieu-jpl has one somewhere with ALL the components and how they are generated. I have a more general for the components our customers should care about: https://lucid.app/lucidchart/invitations/accept/inv_2364907f-f7e3-41f8-8317-05ff0166b77f?view_items=y84_GXILFTJ7%2Cy84_X04RhqJH%2Cy84_41eFhGEk%2Cy84_lrGEepzT%2Cy84_MSAz55go%2Cy84_TyvkpO89%2Cy84_r7wYsRh5%2Cy84_gPSOTB2s%2Cy84_2z1aUVDH%2Cy84_CZ-3-M_G%2Cy84_cdG3_jZv%2Cy84_LqN7xkCL%2Cy84_YU_kjHee%2Cy84_85Zws3Cj%2Cy84_vY0h_RMT%2Cy84_IJebdUoc%2Cy84_EZL8BSY6%2Cy84_L2nWtUm-%2Cy84_2osTO1zZ%2Cy84__.AsTWYr%2Cy84_JC8zbrSL%2Cy84_7klAADxz%2Cy84_Lgi5MTb9%2Cy84_NRgkDRYj%2Cy84_dYWB_~wY%2Cy84_cZeHsMeV%2Cy84_u38l-4DQ%2Cy84_s-usZp2H

But it may even be worth expanding that diagram to include the python client and javalib

pds-api is about the PDS API standard. That is why it is not related to an implementation.

Okay, let's make that clear in the README. I clearly was nosing around in the wrong area! I won't be the first person to make that mistake.

Could you share the stack trace ?

It's linked in the issue above; click/tap the link text "results in a stack trace and perplexing error". Same goes with the link text "even longer stack traces".

The --network pds option was lready there (that is why I am not sure if the documentation I looked at was the one you were refering to).

Maybe I just missed it. Please accept my apologies 🙇‍♀️

One thing that I'm still unclear on is: is it the "PDS API" or the "Registry API"? To me, Registry is just one function of the Planetary Data System, and the PDS may have a server with "validate" capabilities, "ingest" capabilities, etc. Is that coming in the future?

@nutjob4life eventually those microservices would go under the pds-api as well. the registry-api-service is currently being used to encapsulate the pds-api, but I agree that eventually we should probably have a pds-api-service (or some other fun name) that includes the registry search api, ingestion api, transform api, etc.

Thank you @tloubrieu-jpl for your thorough examination of the issue especially while on holiday. Please do not respond to this until you are back! You need a break from this! And thank you @jordanpadams for also leaping to this so quickly. It's obvious I had a lot of confusion and some amount of frustration but I got it working yesterday and some tweaks to the docs might make the next person's go-through smoother. And I'm happy to help with that.

[tloubrieu-jpl]

• [x] The four bullet items suggest there are "libraries" and "stub services" in Python and Java (2×2=4), but…

I created a ticket to clarify the link between api specification and implementations https://github.com/NASA-PDS/pds-api/issues/136

• [x] The "pds-api-javalib" is the "java library" but the "pds-api-pythonlib" is not the "python library" but the python stub service

• [x] (The "pds-api-pythonlib" has no code in it anyway)

I removed the links to the python repository which are not actively developed

• [x] https://github.com/nasa-pds/registry-api-service suggests running mvn clean install sprint-boot:run but this results in a stack trace and perplexing error

This should be solved now, the ticket https://github.com/NASA-PDS/pds-api/issues/123 should also help to set up a development server.

• [x] The same documents give a Docker-flavored solution, but this fails with even longer stack traces

This ticket should also enable to have a reliable docker releases for the registry-app including the API

[nutjob4life]

Thanks for the update @tloubrieu-jpl!

I've also put together this wiki document to help me in future cases to start up the API server. Maybe @ramesh-maddegoda and other future users might find it handy? Maybe not! 😅

[tloubrieu-jpl]

Thanks @nutjob4life I wanted to review all your comments and be sure they are converted into actionable tickets.

I missed the wiki page that you wrote, I share that with Ramesh, as I said in this comment https://github.com/NASA-PDS/pds-registry-app/issues/208#issuecomment-973142174 I am hoping we can make this page much shorter if we organize in a better way our components.

[tloubrieu-jpl]

Now I can close the tickets. Some updates have been made, some other comments are translated into new tickets.

[ramesh-maddegoda]

Thanks @nutjob4life for sharing the wiki. It is helpful!