search

GEMScienceTools / gmpe-smtk

Python and OpenQuake-based Toolkit for Analysis of Strong Motions and Interpretation of GMPEs
GNU Affero General Public License v3.0
44 stars 33 forks source link

Adds support for storing metadata as json #80

Closed g-weatherill closed 6 years ago

g-weatherill commented 6 years ago

Addresses issue https://github.com/GEMScienceTools/gmpe-smtk/issues/79, which raised the idea of using json as an alternative and transferable means of storing the metadata of the strong motion database.

Several changes:

  1. Adds methods to all sm_database objects to dump and/or instantiate the data to a dictionary
  2. Adds methods to sm_utils for deadling with the different OpenQuake rupture object typologies in the case that a rupture surface is present
  3. Adds a general load_database method that allows the user to load a database simply from the database directory path - adopting .pkl or .json depending on what format the metadata file found is.
  4. Tests for json IO

Regarding the performance issues:

  1. It seems as though a json file for the metadata database is about 2.5 times larger in terms of diskspace than its pickle equivalent.
  2. On the tests so far, it is also about 2.5 times slower to build the db from json than to load in the pkl. As anticipated, this is not the fastest or most compact way of storing metadata but it is, for me at least, an acceptable cost to having a transferable file format. In any case, both formats will be supported for the foreseeable future