syntax error in expression in fetch_osc.sh output

[JamesChevalier]

I woke up this morning to a full disk from log output :sweat_smile: (v0.7.56.7)

my fetch_osc.sh command:

nohup bin/fetch_osc.sh `cat db/replicate_id` "http://planet.openstreetmap.org/replication/minute/" "diffs/" >> log/fetch_osc.out &

was dumping this error message into its log file:

bin/fetch_osc.sh: line 147: [[: andard input) matches: syntax error in expression (error token is "input) matches")
bin/fetch_osc.sh: line 158: [[: andard input) matches: syntax error in expression (error token is "input) matches")

those two lines are:

$ sed -n '147p' bin/fetch_osc.sh
  if [[ $REPLICATE_ID -gt $MAX_AVAILABLE_REPLICATE_ID ]]; then
$ sed -n '158p' bin/fetch_osc.sh
  if [[ $REPLICATE_ID -le $MAX_AVAILABLE_REPLICATE_ID ]]; then

my $REPLICATE_ID value looked good:

$ cat db/replicate_id
4196569

I found where $MAX_AVAILABLE_REPLICATE_ID gets its value and manually ran that with an odd result:

$ cat diffs/state.txt |grep -E '^sequenceNumber'
Binary file (standard input) matches

Strangely, it looked fine from the outside:

$ more diffs/state.txt
#Tue Sep 15 12:10:04 UTC 2020
sequenceNumber=4196570
txnMaxQueried=3029111827
txnActiveList=
txnReadyList=
txnMax=3029111827
timestamp=2020-09-15T12\:10\:02Z

But if I edited the file with nano diffs/state.txt it displayed ^@^@^@^@^@^@^@^@^@^@ as the last line in the file.

I edited that line out, stopped processes, rebooted (for computer magic), and started things back up. That seems to have resolved the issue. I'm fine with this fix & things seem to be working properly - figured I should report it here in case there's a bigger underlying issue that you should be aware of.

🌟 Thanks for this great piece of software!

[mmd-osm]

t displayed ^@^@^@^@^@^@^@^@^@^@ as the last line in the file.

This looks like 10 NULL-bytes, which happens to match exactly the difference between the previous and the current state file sizes:

https://planet.openstreetmap.org/replication/minute/004/196/569.state.txt (168 bytes) and https://planet.openstreetmap.org/replication/minute/004/196/570.state.txt (158 bytes)

The syntax error is probably only a side effect of this corrupt state.txt file. Big question is what caused this state file to contain NULL padding bytes towards the end, which clearly don't belong there.

[JamesChevalier]

Maybe it's related to the changes that fixed https://github.com/drolbr/Overpass-API/issues/591

[mmd-osm]

Does your log/fetch_osc.out file include any of the state.txt file sizes? If not, let's add some more logging and see if this can be reproduced.

In fetch_osc.sh, replace:

fetch_file()
{
  wget -nv -O "$2" "$1"
};

by

fetch_file()
{
  wget -nv -O "$2" "$1" 2>> $LOCAL_DIR/fetch_osc.log
};

fetch_osc.log should now include the actual state.txt sizes as returned by the planet osm server. If the file you receive from the server is too large already, it would be an upstream issue (which is somewhat unlikely).

How do you store your diff files on your end? Anything special, like some network filesystem?

(nb: same issue occurred on lz4.overpass-api.de on Oct 15)

[mmd-osm]

Workaround patch is available here now: https://github.com/drolbr/Overpass-API/commit/cfa6ee6f7a1f92e23a267fe1425913ffc7a785cd

[drolbr]

Thank you again for the report. While the deeper cause (how the null bytes got into the state.txt) is still unclear, there is now a permanent workaround as a new minor version