Issue #601 fieldnotes

[hxdimpf]

I am proposing the following design point:

[hxdimpf]

@wrygiel: could you please review this pull request when you have a chance? Thank you!

[wrygiel]

Hi! I'm not much familiar with this. Can you point me to some docs explaining what are field notes, and how they differs from log entries?

[hxdimpf]

Hi! I'm not much familiar with this. Can you point me to some docs explaining what are field notes, and how they differs from log entries?

I am not aware of any documentation but I can quickly explain what it is and also post an example. Some applications on the smartphone, such as: cgeo, Locus Map, Garmin, Looking4Cache, Cachly provide a capability to "offline log" a geocache out in the field, an important function if you don't have a mobile carrier out there. These offline logs end up in a file (or internal data structure) that consists of one or more CSV formatted records like this:

OC1006,2023-11-23T10:17:55Z,Found it,"lorem ipsum ipsum lorem"
OC1013,2023-11-23T10:17:46Z,Found it,"Thx to Chaos"

There are some complications associated with this though. First: It is not standardized. Some tools encode UTF-16, either LE (little endian) or BE (big endian) and there frequently is a BOM (Byte Order Marker), which some tool include and others don't.

For this reason I explilcitly require the client app to convert it on their side to character encoding utf-8 and in addition pack the whole string into base64. The latter is required as the last field in the CSV, the draft log text, may actually span multiple lines in the file and it may contain ASCII control characters such as line feeds, carriate returns, and single and double quotes.

The OCDE website can parse such a file via upload from the browser. The user can then, for each draft log, edit the log text and finally submit it a real log.

With this PR I provide the capability to client applications to directly upload their fieldnotes file into the database as draft logs. Once they are up, there is no difference compared to manual upload via the browser and the filesystem.

[wrygiel]

Isn't it the same as submitting a series of logs via services/logs/submit?

Is the idea to move some work related to processing the file from the app to the server?

[hxdimpf]

Isn't it the same as submitting a series of logs via services/logs/submit?

Is the idea to move some work related to processing the file from the app to the server?

I apologize for being not precise enough: It is not the same as submitting log. Submitting a log is the final transaction that also records a statistics update for the cacher and more importantly makes this log visible in the cache listing.

The upload fieldnotes function puts these logs into a draft state, they will not be commited as real logs yet. There go into a separate table in the database (details). There is a UI in OCDE (as it is on other platforms) to manage these draft logs, e.g. by editing the log text, change time/date, perhaps changinf the log type and then, at the end, press the "submit" button which finally commits a draft log into a real log and this making it visible to othersin the cache listing.

Does this help? Let me know if it doesn't

[wrygiel]

Ah, got it, so a "field note" is a "draft log entry". Yes, I can review it now.

[hxdimpf]

A remark on character encoding of the field_notes parameter: As I already stated, there is no documentation of the format. The layout of the CSV columns is clear, however the character encoding is not. Different applications use different encodings. We could take the approach and say client applications send whatever character encoding they use internally and OKAPI will be smart enough to make us of it. This complicates OKAPI of course. The PR in the current state took the approach: Client, you convert to utf-8 encoding and all is fine. I know that this will cause burden to cgeo as cgeo maintains a fieldnotes file which it typically sends via an API to geocaching.com. Ideally cgeo would want to send the same data over OKAPI, least amount of effort on cgeo's side.

However, and for whatever reason cgeo uses an obscure encoding, which I believe is not even standardized. I also know that cgeo hast changed that a few years to what it is now. For reference I include a cgeo fieldnotes file header in hex such that you can draw your own conclusion:

00000000: 4f00 4300 4400 3300 4100 4400 2c00 3200 O.C.D.3.A.D.,.2.

The only thing that's clear is that it is a 16-bit encoding, however:

enca -L none 20231127092821.txt 
Universal character set 2 bytes; UCS-2; BMP
  LF line terminators
  Byte order reversed in pairs (1,2 -> 2,1)

It seems to be UTF-16LE without BOM.

So what do we do, accept this format? Tell cgeo to encode in UTF-8? Be very smart and accept it all whatever it is?

Update: I generated a very short line, a single log. Such that I could paste it into the OKAPI browser:

head -1 20231127092821.txt
OC1010,2023-11-27T08:27:57Z,Write note,"Thx to Borstelmann  for 'Lindenallee"

Which is in hex:

head -1 20231127092821.txt |xxd
00000000: 4f00 4300 3100 3000 3100 3000 2c00 3200  O.C.1.0.1.0.,.2.
00000010: 3000 3200 3300 2d00 3100 3100 2d00 3200  0.2.3.-.1.1.-.2.
00000020: 3700 5400 3000 3800 3a00 3200 3700 3a00  7.T.0.8.:.2.7.:.
00000030: 3500 3700 5a00 2c00 5700 7200 6900 7400  5.7.Z.,.W.r.i.t.
00000040: 6500 2000 6e00 6f00 7400 6500 2c00 2200  e. .n.o.t.e.,.".
00000050: 5400 6800 7800 2000 7400 6f00 2000 4200  T.h.x. .t.o. .B.
00000060: 6f00 7200 7300 7400 6500 6c00 6d00 6100  o.r.s.t.e.l.m.a.
00000070: 6e00 6e00 2000 2000 6600 6f00 7200 2000  n.n. . .f.o.r. .
00000080: 2700 4c00 6900 6e00 6400 6500 6e00 6100  '.L.i.n.d.e.n.a.
00000090: 6c00 6c00 6500 6500 2700 2c00 0a         l.l.e.e.'.,..

And guess what: The OKAPI browser's signature was also rejected by the OKAPI server:

The remote server returned an error: (401) Unauthorized.
{"error":{"developer_message":"Invalid signature.","reason_stack":["invalid_oauth_request","invalid_signature"],"status":401,"more_info":"https:\/\/opencaching.pl\/okapi\/introduction.html#errors"}}

So this tells me (what I actually new ahead of time), that this input format you see above (in hex) is actually garbage, however, garbage I can't make cgeo change.

I will do one more test with PLAINTEXT signature on this payload but I am almost sure it will fail too as that character encoding is not understood by PHP and it can't be converted as I have already tried:

 $decodedFieldNotes = base64_decode($field_notes, true);
        if ($decodedFieldNotes === false) throw new InvalidParam('field_notes', "Input data is not properly base64 encoded.");

        $detectedEncoding  = mb_detect_encoding($decodedFieldNotes, 'UTF-8, UTF-16', true);
        $utf8String  = mb_convert_encoding($decodedFieldNotes, 'UTF-8', $detectedEncoding);

[SammysHP]

However, and for whatever reason cgeo uses an obscure encoding, which I believe is not even standardized.

The uncommon encoding has historic reasons. It was specified/implemented that way a long, long time ago together with Garmin GPS receivers and hasn't changed since then.