search

ai2cm / fv3config

Manipulate FV3GFS run directories
Apache License 2.0
1 stars 0 forks source link

Encode bytes data using base64 #102

Closed nbren12 closed 4 years ago

nbren12 commented 4 years ago

Currently bytes assets are serialized to yaml using the specialized !! notation of yaml.safe_dump. This is how yaml encodes a bytes object:

In [1]: import yaml

In [3]: yaml.safe_dump(b"hello world")
Out[3]: '!!binary |\n  aGVsbG8gd29ybGQ=\n'

It looks like it is using base64 encoding to do this.

This will probably not be interoperable with non-python yaml parsers, so this PR introduces our own encoding strategy for bytes assets returned by get_bytes_asset_dict.

This change is not backwards compatible since old fv3config.yml's with bytes assets will not work with this new code. I don't think we will run into any problems since we have never actually tried to re-run past runs with old fv3config.yml's, and the bytes_asset_dict feature is relatively recent.

Nonetheless, we may want to introduce some kind of versioning strategy for serializing/deserializing fv3config.ymls if only to warn the user if they try to load an out of date fv3config.yml.

nbren12 commented 4 years ago

Actually, the !!binary syntax is part of the yaml specification. I am closing this PR.