mac export large data failure

[caver456]

This was first noticed during another user's testing of the mac installer (#27). Export fails with the red line 'Unexpected disconnect' when all features from a certain sartopo page are exported. The log file just ends in the middle of the 'request=' data line. Windows export works fine on the same data.

[caver456]

Using #20, the log file for the failed export looks like this:

INFO : GPSIO Host invoked at Tue 21 Dec 2021 20:12:15
INFO : Python=3.8.2 (default, Dec 21 2020, 15:06:04)
[Clang 12.0.0 (clang-1200.0.32.29)]
INFO : host version=1.2.0
INFO : .ini file=/Library/GPSIO/gpsio-host.ini
INFO : GPSBabel executable=/Applications/GPSBabelFE.app/Contents/MacOS/gpsbabel
INFO : data transfer chunk size=100000
INFO : platform=darwin
INFO : arguments:['/Library/GPSIO/gpsio-host', 'chrome-extension://cbpembjdolhcjepjgdkcflipfojbjall/']
DEBUG : request_length=int:92581
DEBUG : request=str:{"source":"page","type":"gpsio","id":232806,"cmd":"export","target":"garmin","data":"<?xml version=\"1.0\"?>\n<gpx xmlns=\"http://www........[very long line; the last characters on the line are:].......541437\"/><trkpt lat=\"39.3432883
CRITICAL : Uncaught exception
Traceback (most recent call last):
  File "gpsio-host.py", line 560, in <module>
  File "gpsio-host.py", line 197, in Main
  File "json/__init__.py", line 357, in loads
  File "json/decoder.py", line 337, in decode 
  File "json/decoder.py", line 353, in raw_decode
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 85 (char 84)

[caver456]

So what causes the unterminated string? Note from the traceback that it occurs on the json.loads line - not on the logging.debug line that causes the incomplete log entry.

    193  logging.debug("request="+type(request).__name__+":"+str(request))
    194
    195
    196  # validate and parse the request
    197  rq = json.loads(request)

Still suspicious that it's a stdin issue. The sent data looks fine in the service worker inspector console in Chrome. The request= line in the log (for smaller data) is identical on mac vs Windows (except for id which is an unquoted 6-digit integer on both).

(I thought request_length was one greater on mac than Windows, but that's not accurate: that was because removeNumbers was 'false' on mac (5 characters) at the time, unchecked in the popup, but 'true' on Windows (4 characters). Checking the box in the popup on mac results in identical request_length as Windows.)

[caver456]

Still need to find out if this is a fair comparison: the request data printed in the log file request= starting with { thru the end of the line) is 65538 characters. Interesting number... (2^16=65536) - so only the first 2^16 bytes from stdin are making it to the program, even though 92580 were requested...?

[caver456]

Did some googling and RTFM about buffering in the python io docs. For some reason, when run from the command line, stdin of 92580 bytes works fine, but when run as a pyinstaller executable as the extension native host, stdin gets lopped off at 65k.

The code was trying to open stdin unbuffered:

sys.stdin=os.fdopen(sys.stdin.fileno(),'rb',0) # unbuffered

RTFM:

buffering is an optional integer used to set the buffering policy. Pass 0 to switch buffering off (only allowed in binary mode), 1 to select line buffering (only usable in text mode), and an integer > 1 to indicate the size in bytes of a fixed-size chunk buffer. When no buffering argument is given, the default buffering policy works as follows:

Binary files are buffered in fixed-size chunks; the size of the buffer is chosen using a heuristic trying to determine the underlying device’s “block size” and falling back on io.DEFAULT_BUFFER_SIZE. On many systems, the buffer will typically be 4096 or 8192 bytes long.

Based on that, maybe we DO want it to be buffered, since hopefully the chunks are less than 65k.

Sure nuff, this seems to work - need to test more before closing the issue:

sys.stdin=os.fdopen(sys.stdin.fileno(),'rb') # standard buffering

[caver456]

Tested on 2.4MB gpx: export to gpx, and export to gpsio, created identical output except for:

• gpsio added the trackextensions xmlns in the header
• gpsio pruned numbers from exported track names when number was prepended by text and space, as expected