extracting ascii from dat files

[ksterne]

Hey all, I'm trying to figure out why two copies of the same dat file seem to have a different checksum. If I had rawacf files (or higher-level data products), I would do a dmapdump on the files into two files and run diff on the two ascii files. Is there a way do to this currently with dat files? Maybe I'm mostly looking at @egthomas as I think you've worked with dat files the most, but maybe others here have so I've posted my question here.

I think it might be easy to create a binary to do this, but I didn't want to re-invent the wheel.

[ksterne]

And I've tried out cmpraw, but it seems to only print the timestamps of the records without giving too much detail. I may use that as a starting point, but any other ideas here are welcomed.

[mtwalach]

Can you use dattorawacf and then compare the dmapdump outputs?

[ksterne]

That's one possible route I thought of, but then I'm not sure how much dattorawacf might modify the original values from the 2 files. Certainly the origin.time values would differ unless I try to run them at the same time. Thanks for the suggestion though!

[mts299]

Any luck on this @ksterne ?

[mts299]

@ksterne Did you figure this out?

[ksterne]

I haven't been able to figure this out, though the need has decreased for now I think (or more I haven't worked on the issue where I needed this kind of tool in a while). I think the solution is likely to create some type of tool similar to map_ascii where the contents of the file are "uncompressed" out of the binary format. Does that generally sound like a way forward to solve this kind of thing?

If so, I think this could be the start of a bigger kind of project for creating tools that could do a similar job. Most of our data formats are taken care of (all the ones using dmap), but just like the mapascii tool, there could be some other conversions to formats or data-levels that people are looking for. Any thoughts? If not, maybe it's at least a two-off need for being able to examine the contents of dat and fit files in a human readable format.

[egthomas]

@ksterne cmpraw does exactly what you're asking for - it compares the values of every field and array between two dat files and prints out any differences. If the only thing being printed on each line is a time stamp then that means there are no differences:

https://github.com/SuperDARN/rst/blob/master/codebase/superdarn/src.bin/tk/testing/cmpraw.1.6/cmpraw.c#L112-L275

[ecbland]

@ksterne Can we close this issue? It sounds like cmpraw and the new datdump are very helpful here.

[ksterne]

Let me review this in the next couple of days. I'm betting it can be, but let me review to be sure.

[ksterne]

Closing this now since #376 has been merged which takes care of this need. Thanks again for working on creating this! This will be helpful when looking at the contents of dat files and not just comparing 2 files as cmpraw does.