search

nasa / cFE

The Core Flight System (cFS) Core Flight Executive (cFE)
Apache License 2.0
408 stars 200 forks source link

Table decompression #407

Closed onurturhan closed 4 years ago

onurturhan commented 4 years ago

Is your feature request related to a problem? Please describe. Compressed tables could save space in NV-RAMs.

Describe the solution you'd like The decompression feature (implemented already for apps) can be extended to tables.

Describe alternatives you've considered None.

Additional context N/A

Requester Info Onur TURHAN

onurturhan commented 4 years ago

table_decompression.diff.txt

jphickey commented 4 years ago

This will require some community discussion - this decompress for apps is being considered for removal as it has been identified as a (possibly) un-required feature.

The problem is that the decompress happens to a RAMDISK so you simply trade a slight NVRAM savings for significant increase in RAM use, as the table is now effectively loaded into RAM twice. In many flight systems the RAM is just as constrained, if not more so, than the NVRAM storage.

Can you identify a specific use case for this (i.e. a flight computer where the NVRAM is constrained such that this would be a requirement)?

skliper commented 4 years ago

Note that the decompression functionality is transitioning out of cFE to a library (likely called FS_lib), so the FM app will retain it's capability to decompress (and actually untar is also getting added). The dependency of cFE on decompress is being removed, so "automatic" decompression as part of load isn't supported.

The plan is to continue to support decompress with FM, then you can load the app/lib/table. It just won't be automatically done within cFE. The use case for us is to handle transferring many large tables to a cFS based system as tar.gz, then decompressing and untaring on board.

A major motivation of moving decompress (and untar) into a library is to make it easier for users to replace with whatever decompression algorithm they'd like.

onurturhan commented 4 years ago

There could be two requirements actually,

  1. flight computer where the NVRAM is constrained. Yes, actually I encountered two times NVRAM/RAM size rate as 1/4, so at least projects that I work have such a situation.

  2. size reduction important for also uplink. There are requirements for LEO satellites like updating all applications in one pass, so this requires more uplink rate if there is no compression.

As an example 270KB CF(CFDP App) (rtems/sparc,stripped) task could be zipped to 70KB (26% compression rate). This is usually better for tables due to repetitive parts like Space Packet Header (ex;sch_def_msgtbl). In projects, I have encountered an average ~1/10 compression rate for tables. Also, ATS and RTS tables loaded daily to some spacecraft, so 1/10 compression is very good for effective operational time usage.

On the other hand, implementing this feature with FM will cause at least three constraints or situations in order to load compressed lib, app and tables.

IMHO, FM app should have compression and decompression features. Especially, file compression will be very useful for DS app output files. However, compression & decompression functions can be implemented in OSAL(FS), since some operating systems have these features inherently (posix,rtems). There could be some flag to disable this feature for unsupported RTOS. ES and TBL apps could use these functions to load zipped app, libs, tables, if RTOS supports. FM can use OSAL functions or any other loaded comp-decomp library like FS_lib with hooks or options.

skliper commented 4 years ago
  • FM app and its tables (or FS_lib) should not be compressed.

True (along with the core).

  • FM app (or FS_lib) should be loaded before any other compressed tasks (with related tables).

True.

  • ES app should send a command to FM app before loading apps, or TBL app should send a command to FM app before loading the tables. Or maybe ES & TBL will use FS_lib functions after loading FS_lib. (Assumption: cfe_es_startup.scr is used to load all apps.) So circular dependency between CFE-CORE apps and FM app or FS_lib.

Plan is FM would need to decompress before the load command to ES. ES would not have any linkage to decompress or FM. Could be a scripted set of startup commands. Compressed file system is also a valid approach for operating systems that support it(below the cFS level). There's other approaches outside cFS if there are unique system restrictions (decompress into ram as part of boot allows you to compress everything, etc).

IMHO, FM app should have compression and decompression features. Especially, file compression will be very useful for DS app output files.

Decompress is being maintained in FM app. Compression (as a library addition) could certainly be considered, and I encourage open source contribution of a cFS/Flight friendly implementation.

However, compression & decompression functions can be implemented in OSAL(FS), since some operating systems have these features inherently (posix,rtems).

Note that built in support is frequently not flight software friendly (recursive, unbounded timing, dynamic memory use, non-portable, etc). There is the shell command from ES already if you want to evoke built in tools on a supporting OS.

There could be some flag to disable this feature for unsupported RTOS. ES and TBL apps could use these functions to load zipped app, libs, tables, if RTOS supports. FM can use OSAL functions or any other loaded comp-decomp library like FS_lib with hooks or options.

Planned approach is to abstract via FS_lib, and maintain a common set of API's and allow projects to easily swap out for other implementations.

skliper commented 4 years ago

Note there are projects that as part of boot (or any time prior to loading cFS) decompress the entire file system (or parts like image/data/tables/calibration/etc). Typically using whatever built in or custom decompression a project desires. Allows for maximum compression, project customization, and independence from cFS. This is the preferred approach vs decompressing at load within ES. For anything loaded after starting cFS, FS can be used to decompress.