search

pywr / hydra-pywr

GNU General Public License v3.0
1 stars 2 forks source link

Data being stored in s3 has a bytestring in it, which will likely cause issues #35

Closed knoxsp closed 2 weeks ago

knoxsp commented 2 weeks ago

Data being stored now looks like this:

{ "data": { "url": "s3://pywr-results/b'\x89\xa2T\r\xff\x0cB5\xaf\xa2\xf0\x94`/%\x9d6\xbb>\xa1\xa4\xfc\xebss\xaf\\\xfe\xb28\xba)'/simulated_volume.h5", "group": "YavanDam" } }

Notice that the ID has a byttestring in it, instead of a standard string. This should be changed to be more decipherable.

pmslavin commented 2 weeks ago

This is coming from the hmac.digest on L160...

    hashkey = hashlib.sha256(randbytes(56)).hexdigest().encode('utf-8')
    self.s3_path = hmac.digest(hashkey, str(self.scenario_id).encode('utf-8'), hashlib.sha256

So if a string of some hex is acceptable we can just call .hex() on the bytes instance it returns. The url above would become...

s3://pywr-results/89a2540dff0c4235afa2f094602f259d36bb3ea1a4fceb7373af5cfeb238ba29/simulated_volume.h5
knoxsp commented 2 weeks ago

Looks good. Can you put that in a suggestion?

pmslavin commented 2 weeks ago

Added this on the branch here.

Both the uses of s3_path after that (L314, L712) are as strings so no-longer being bytes should be fine.