search

conda-forge / eccodes-feedstock

A conda-smithy repository for eccodes.
BSD 3-Clause "New" or "Revised" License
4 stars 18 forks source link

Enable MEMFS (memory based access to definition/sample files) #26

Open kynan opened 7 years ago

kynan commented 7 years ago

As of 0.16.0 ecCodes supports memory based access to definition/sample files:

From this version the definition and sample files used by ecCodes engine to decode all kind of messages can be built in to the library (embedded) providing a performance improvement by significantly reducing access to the filesystem.

This feature is called MEMFS (MEMory FileSystem) and can be enabled at configuration time (when you invoke cmake) by:

cmake /path/to/src -DENABLE_MEMFS=ON

This will take longer to compile and the resulting library will be larger but whenever a definition file (.def or .table) or a sample file (*.tmpl) is accessed, its contents will be fetched from memory and no access to the file system will be made.

If you invoke the codes_info tool with this configuration, the paths reported will contain the prefix "/MEMFS/".

Please note that once ecCodes is built in this mode, any subsequent changes to the installed definition/sample files will NOT be picked up

How do does everyone feel about enabling this? @kmuehlbauer @pelson @ocefpaf

kmuehlbauer commented 7 years ago

Please note that once ecCodes is built in this mode, any subsequent changes to the installed definition/sample files will NOT be picked up

How do does everyone feel about enabling this? @kmuehlbauer @pelson @ocefpaf

I'm very much open, if this improves speed significantly. But there is a drawback. Users won't be able to add/change definitions and tables in their installations. This could turn the package useless in some environments.

kynan commented 7 years ago

True, should this be a dedicated feedstock then?

kmuehlbauer commented 7 years ago

Maybe we can use a dedicated branch? Maybe @ocefpaf has an idea.

ocefpaf commented 7 years ago

Maybe @ocefpaf has an idea.

Branches are probably better, but we would need to disambiguate the name with a prefix or something similar.

kynan commented 7 years ago

variant/memfs? @ocefpaf

kynan commented 7 years ago

@kmuehlbauer @ocefpaf I actually want to try this to work around a runtime failure I'm currently trying to debug rather than for performance. If we can agree on a branch naming convention I'll give it a go.

cc @shahramn

kmuehlbauer commented 7 years ago

@kynan

You could try with a PR first, to see if this will build correctly. Then @ocefpaf might help us to get the package prefixing correct. Then the PR can be diverted to the new branch.

ocefpaf commented 7 years ago

You could try with a PR first, to see if this will build correctly. Then @ocefpaf might help us to get the package prefixing correct. Then the PR can be diverted to the new branch.

Sounds like a plan.

kynan commented 7 years ago

Done! #28

kmuehlbauer commented 3 years ago

With #127 it is available on windows. Users can add local tables/definitions. Question: Is it only relevant for windows or does also linux/mac builds profit, @shahramn?