search

nasa / fprime

F´ - A flight software and embedded systems framework
https://fprime.jpl.nasa.gov
Apache License 2.0
10.05k stars 1.3k forks source link

Data Product Catalog prototype #2667

Closed timcanham closed 5 months ago

timcanham commented 6 months ago
Related Issue(s) https://github.com/nasa/fprime/issues/1713
Has Unit Tests (y/n) y
Documentation Included (y/n) y

Change Description

This is a prototype demonstration of Svc/DpCatalog It was used to demonstrate an early release of data products in conjunction with FPP updates and DpManager/DpWriter

Svc/DpCatalog is not complete yet - this version is to capture the demo and allow users to play with it.

Rationale

Get a consolidated set of data product features int he baseline.

Testing/Review Recommendations

The demo can be run and experimented with.

Future Work

Svc/DpCatalog work will be continued with further releases.

timcanham commented 6 months ago

Requested @bocchino for changes made to Svc/DpWriter.

Joshua-Anderson commented 6 months ago

Is storing transmit state (UNTRANSMITTED/PARTIAL/TRANSMITTED) in the data product header the long term term plan?

It seems like this would force us to re-write data products once we start transmitting them, which could be a bit of a circular dependency. It also could get a bit confusing because the onboard DP file checksum would change after you start transmitting it.

Did we consider having DpCatalog own data product transmit state instead? (I.E. have a catalog file with a list of data products and their transmit states)

This could also provide a bit more flexibility because different projects may want to track data product transmit state differently (I.E. track not just that a data product is partial but how much of the file was sent, or which portions of the file need to be retransmitted).

timcanham commented 5 months ago

@Joshua-Anderson The plan is to modify the header as we process the data products, i.e. DpWriter writes it as UNTRANSMITTED, then as DpCatalog starts downlinking it will change to PARTIAL, then TRANSMITTED when done. I'm not sure I see the circular dependency part, so elaborate on that for me.

You are correct that having a separate DP state file (or other variants on the idea) is an option and @bocchino and I discussed it, but I've always preferred files to be self-contained in their state so that the DpCatalog can be simpler in that it discovers state in the files and doesn't have to keep track of things in two places. Just a design preference.

bocchino commented 5 months ago

The changes to DpWriter look good to me! I just had a couple of comments.