search

MetPX / sarrac

C implementation of (a subset of) Sarracenia (large scale file transfer utility)
GNU General Public License v2.0
4 stars 1 forks source link

non UTF8 header seen by sarrac on flow_test (cpump) #133

Open petersilva opened 10 months ago

petersilva commented 10 months ago

running flow test suite, from cpump_pelle_dd1_f04_01.log

2023-12-05 19:19:00,582 [WARNING] skipping non UTF8 headers: amount: 7, this one: 3, kind:73
2023-12-05 19:19:00,582 [INFO] received: { "pubTime":"20231206001857.646", "baseUrl":"https://hpfx.collab.science.gc.ca", "relPath":"/20231206/WXO-DD/.hello_weather/observations/03/03048.1.OBS.PUB.FR.SPX~20231206000500-20231206020500-1427", "topic":"v02.post.20231206.WXO-DD.hello_weather.observations.03", "identity":{  "method" : "md5", "value" : "aOhJn7exRUsa1TAq1lMZlg=="  } , "mtime":"20231206001900.53", "atime":"20231206001900.53", "mode":"0666", "size":"96940", "to_clusters":"DDI,DDSR", "sundew_extension":"DMS:CMC:WXR-TRANSCODER-OBS-DATAMART-IVR:AUDIO"}
petersilva commented 10 months ago

so ran into this issue where all of the shim tests failed... and then went back trying to identify when they fail and all the old version fail also.

I eventually found that the problem was in the python implementation, there is a commit that changed how sftp urls are handled, and that broke the shim tests... somehow... was busy, did something else... have to repeat the git bisection to find the exact commit again. but at least it explains why all tests were failing. It was not a change in sarrac that broke things.