New serial download code

[steko]

Hello, this is a dirty work-in-progress pull request that I'm sharing just to let @psolyca and all @totalopenstation/contributors comment. It started as a small PR to add "autosave" but since all serial download code is very entangled, I realized that it was impossible to develop without proper tests ... and I'm stuck with tests as well.

[steko]

See https://github.com/totalopenstation/totalopenstation/issues/155 which is the main issue with the current serial download code.

[psolyca]

Ok, just by reading your code, it seems correct. I can not test it yet.

BTW, what is the result of the test ? Also, is the autosave file filled ? ExitStack should handle that at the end of the process.

It should be better to use a real station to test it ! Otherwise, 2 computers, one send data and another read it ! Just to be more "accurate"

[steko]

The autosave is filled, yes.

But I'm having issues with the serial buffer size (4096), I can't write more than 4096 bytes.

With the Thread involved, I have trouble using the loop:// device to test writing and reading without hardware involved. That is the only way to get proper automated testing, I think.

How would you go at transmitting data from one computer to another?

[psolyca]

Between computers, in https://pyserial.readthedocs.io/en/latest/url_handlers.html, maybe you can use rfc2217:// or socket:// where you can choose a host and a port.

For Thread, have you look at https://pyserial.readthedocs.io/en/latest/pyserial_api.html#module-serial.threaded ? Even if it is experimental.

But I'm having issues with the serial buffer size (4096), I can't write more than 4096 bytes.

Write to the loop ? Why do you read 4096 bytes, testdata.read(4096) ? On the Python documentation, testdata.read() should return the entire file, then the read_block = self.ser.read(self.ser.in_waiting) should handle each iteration !

EDIT : I find something, default behaviour of open is using a buffer of 4096 or 8192 bytes. But if you use buffering option in open you can change it to disable buffering. So with open("sample_data/zeiss_elta_r55/zeiss_elta_r55-REC_500.tops", "rb", buffering=1) as testdata: should did it.

[steko]

EDIT : I find something, default behaviour of open is using a buffer of 4096 or 8192 bytes. But if you use buffering option in open you can change it to disable buffering. So with open("sample_data/zeiss_elta_r55/zeiss_elta_r55-REC_500.tops", "rb", buffering=1) as testdata: should did it.

I had not seen your edit to this comment, sorry about the delay! I'm gonna follow your suggestion about buffering and report here.

[psolyca]

Sorry @steko it'ss still a draft or not ?

[steko]

Yes still a draft.