search

jcsambangi / bme590final_backend

Back end functionality for the HIE sensor management tool.
MIT License
1 stars 0 forks source link

Data and Encoding #15

Open termao6 opened 5 years ago

termao6 commented 5 years ago

For the data that we're reading in from the binary file, do we have to save it in the DB as an encoded base64 characters? Or can we store the raw f.read() data?

Also, when I read in the data as binary, I'm getting odd character such as b'\t' and b'!'. The first few bytes I'm reading look like this: b'\x00'b'\x00'b'\x00'b'\xe8'b'\x03'b'\x00'b'\x00'b'\x0c'b'\x00'b'\x00'b'\x00'b'\t'b'\x00'b'\x00'b'\x00'b'\x00'b'\x00'b'\x00'b'\x00'b'!'b"'"b'\x00'b'\x00'b'\x00'

^ That is reading the file one byte at a time. Calling f.read() also gives me this funny piece near the beginning: x00!\'\x00\x00\x00\x00\x00\x00s\x01\x00\x00...06P\x7f\xd0 with the ellipses actually in there.

And when I encode it to base64, I get this: b'YwAAAOgDAAAMAAAACQAAAAAAAAAh at the start

I'm not sure what's going wrong... is the data bad? Am I reading in the binary wrong? (I'm using "with open(filepath, "rb") as f:") Is something (also?) going wrong in the encoding?

[will update as I figure this out]

termao6 commented 5 years ago

Nevermind, I think the data does just look funny and everything works.

I checked against the uint8 encoding (decoding?) that Dr. Luck's matlab file uses. All files start with 99 0 0 in uint8. This matches the base64 encoding(decoding?) that I got according to my manual decoding.

Still don't understand why the binary (technically hex) looks weird, but whatever, it works. Though if someone has an answer, I'd like to know.

termao6 commented 5 years ago

Actually, why is my decoding of the base64 so weird? And the raw data, for that matter. I'm using 'rb' to read the file...

raw file reading: 1\x9e7\x00\x00\x80\x7f\xf0\x7f\xc0\x80c\x80\xd0\x80j\x7f\x07\x19\x94\x01\x80\x06\xf0\x7f\xd0~@\x81\xae\ b64: b'YwAAAOgDAAAMAAAACQAAAAAAAAAhJwAAAAAAAHMBAAAbAAAAQAEAAMi decoded: b'c\x00\x00\x00\xe8\x03\x00\x00\x0c\x00\x00\x00\t\x00 uint8 array: 99 0 0 0 232 3 0 0 12 0 0 0 9 0 0 0 0

code: with open(filepath, "rb") as f: data_raw = f.read() b64data = base64.b64encode(data_raw) print(b64data) decoded = base64.b64decode(b64data) print(decoded)