updated to add dcs and decoding some Sutron types, still todo type C & D.

[twcutter]

I've got dcs working for type B and human readable types via Pseudo-Binary additons. The dcs/sutron.cc is a work in progress to add the less common type C and type D formats of GOES DCS/DCP data. It is described in pdf documents in Sutron, now ott.com products. I'd like to also add CRC-16 and CRC-32 to work like Taylor's Python setup. For check summing the File Header (32) and Block Headers (16). There is a dcs_handler added from work of others, also.

[twcutter]

I've updated the sutron.cc decoder to include Sutron type B,C,D. This completes the decoder. There are other types that are present and unknown how to decode generically. All the documented "Sutron" stuff is done for the decoder.

I considered previously in adding a CRC check which would need CRC16 and CRC32. I believe that what is sent as CRC from the DCP Stations is Diagnostic for the Station maintainer and the serial UART of the sending devices. The DCP station maintainers may be interested in the CRC values sent from the station? The values for hobbyists, I'll just report in output.

[twcutter]

Andrew, I’ve put the change in locally and I’ll see how it works. I don’t get many Missed Messages to even know what I have is correct. The idea makes sense. The blocks may change one day? If I get something back like expected results. I’ll submit the change.

• Tom

On May 6, 2022, at 12:52 PM, Andrew Chin @.***> wrote:

@eminence commented on this pull request.

In src/dcs/dcs.cc https://github.com/pietern/goestools/pull/139#discussion_r867176662:

+ +

• while (nread < len - 4) { // 4 for ending header CRC32 per spec
• blocks.resize(pos + 1);
• // 1 byte contains blocks ID
• {
• constexpr unsigned n = 1;
• ASSERT((len - nread) >= n);
• memcpy(&blocks[pos].blockID, &buf[nread], n);
• nread += n;
• }
• // Skip over Missed Messages Blocks (blockID == 2 as of January 21, 2021)
• if (blocks[pos].blockID != 1) {
• nread += 28; // Size of Missed Message Block (2(blocks len) + 24(header) + 2(CRC16)) Skipping over 28 bytes is correct for messages with blockID=2, but it would be better to read the next 2 bytes to calculate the block length in order to figure out exactly how many bytes to skip

— Reply to this email directly, view it on GitHub https://github.com/pietern/goestools/pull/139#pullrequestreview-965133214, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACTDLQZE3AML3W5JCHHNB43VIV2BBANCNFSM5UTVQU7A. You are receiving this because you authored the thread.

[eminence]

Hi Tom,

I've personally never seen any blocks other than 1 and 2, but the HRIT_DCS_File_Format_Rev1 pdf explicitly recomments checking the block length (section 3.2)

To support future enhancements while maintaining backward compatibility, this proposed HRIT file format utilizes a block identifier (BLK_ID) immediately followed by a block length (BLK_LNG) field. When implementing the file processing code, systems should be designed to look at the block identifier to determine the type of data in the block and the appropriate handling. If the BLK_ID is not a value the code recognizes or supports, the code should then use the length field to skip over this data.

[twcutter]

I've added a few type D hash checks which look like data to sutron.cc. The type B,C & D is only suppose to be 1,2,3,4 for the second digit of data. But then there is real data showing other characters sent with type D. I've also commited @eminence suggested change of missed blocks in dcs.cc. It makes sense to do so.

[eminence]

The PDF files should probably be added in the docs/files/ folder instead of being in the same folder as the source files

[elafargue]

It would be worth squashing all of these commits into one larger commit once this is ready for merge, this will be a lot cleaner.

[twcutter]

I’ve not been happy with the sutron.cc http://sutron.cc/ decoder. It is the last piece I’ve written. In studying the following code: https://github.com/opendcs/opendcs/blob/master/src/main/java/ilex/util/PseudoBinary.java https://github.com/opendcs/opendcs/blob/master/src/main/java/ilex/util/PseudoBinary.java

I’ve found there are two ways of going in decoding. One is to assume that all values are unsigned numbers. That is how I wrote it last week. The other is to assume that the battery voltage and the 18 bit precision numbers are signed. I have not had enough time to decode existing data to see which way was correct in general. I’ve modified sutron.cc to use signed numbers as a default. I think this is the correct approach. In decoding messages today, it appears nothing is really consistent. It takes some information from each DCP sites specifically to chose the correct path. I think what is present now is the closest to a generic solution and I’m done, for making changing to it for now.

The USGS data of water probably does not have negative values. Stream gage height and water temperature (flowing water) would be positive. Then there are sensors which I’m not familiar which could have negative numbers. Like air temperature. I changed my direction, negative numbers are possible. Let’s see how far the decoder takes it.

• Tom

On May 10, 2022, at 1:41 PM, Edouard Lafargue @.***> wrote:

It would be worth squashing all of these commits into one larger commit once this is ready for merge, this will be a lot cleaner.

— Reply to this email directly, view it on GitHub https://github.com/pietern/goestools/pull/139#issuecomment-1122840014, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACTDLQ55JUCDVI2VQGDQ2JDVJLCW3ANCNFSM5UTVQU7A. You are receiving this because you authored the thread.

[twcutter]

I didn't convert a missing value which is designated in GOES data as three slashes /// . This will turn into some form of negative number. At least through observation. Big negative numbers are either a test sent, or I'm not familiar with a sensor which would provide a hugh negative number? I'd expect negative numbers to be small. Large negative shows a conversion problem?