search

datajoint / datajoint-matlab

Relational data pipelines for the science lab
MIT License
42 stars 38 forks source link

jobs key hash formats differ between matlab and python #349

Closed ixcat closed 3 years ago

ixcat commented 3 years ago

Bug Report

Description

Datajoint python vs datajoint matlab use different key hashes for job reservations. This means that each language will not honor job reservations from the other language when attempting to compute the same keys on the same table.

Reproducibility

Since the recommended practice is to never implement workflows where the same computation can be performed in either language, this incompatibility does not cause job reservation issues.

Expected Behavior

Ideally, job reservations should be honored across languages. However, in practice, the recommended practice is to avoid implementing the same jobs in multiple languages in the same workflow and the issue of incompatible job reservations does not arise.

Screenshots

$ egrep -in '(md5|sha-256)' ./datajoint-matlab/+dj/+internal/hash.m  ./datajoint-python/datajoint/hash.py 
./datajoint-matlab/+dj/+internal/hash.m:6:    dj.lib.DataHash(data, struct('Input', inputType, 'Method', 'SHA-256', 'Format', 'base64')), ...
./datajoint-python/datajoint/hash.py:13:    hashed = hashlib.md5()
./datajoint-python/datajoint/hash.py:25:    hashed = hashlib.md5(init_string.encode())

Additional Research and Context

given the corner case nature of this & going against recommendations, this is probably not a very 'required' fix. filing mainly for documentation purposes.

ixcat commented 3 years ago

resolving as wontfix per discussion/above notes; if people do experience issues with this we are open to revisiting - please reopen with appropriate details.