Long exchanges truncated

[zcsahok]

Exchange input field allows to enter up to 25 characters.

After logging the QSO the exchange is truncated to 14 characters.

Same in the log file:

[dl1jbe]

I assume you were in contest mode while testing.

The behavior is due to limitations in the TR log format we inherited from the original program. That format reserves a varying amount of line space for recording of multipliers and points depending on the contest/dxped/qso mode you use.

The exact amount which gets recorded is determined in makelogline.c in prepare_specific_part.

I agree that we need a better solution. Best would be to drop the old log file format and switch to a more flexible one - I would suggest a database related approach here - but other schemes may be possible. Downside is that it breaks compatibility with older log version completely.

As it looks we have quite some serious changes to make

• log format
• handling of configuration storage and usage
• rework of point and multiplier scoring logic
• resizeable layout in horizontal and vertical direction

which may all break compatibility in a drastic manner.

Therefore I would suggest starting a discussion about a new TLF-2.xx version. Once the goals are clear we could prepare one last TLF-1.x.x version with all the features and fixes done in since TLF-1.4.1 and than start over to work on the new version.

[zcsahok]

For me limiting the exchange input length to 14 chars would be perfectly fine. At least it must be consistent.

[dl1jbe]

Good then let us keep the issue open to remind us to fix it.

[N0NB]

Let me offer this, if a major rewrite of the log file format is to be done, do we want to retain the simple flat file format or go to something like an SQLite database?

A simple flat file format is easy to work with in any text editor but it is fragile in the sense that great care needs to be take to assure its integrity throughout the program. Using an SQLite database may protect against this somewhat though it is not as strict (as I recall) enforcing data types as Mariadb or Postgresql, for example, but does not require a DB administrator either as everything is kept in one file. SQLite has bindings for many languages so external log analysis tools could be written fairly easily.

On the other hand, it may be easier to recover a corrupted flat text file than an SQLite db from a corrupted SD card, etc. (often times in those cases nothing is recoverable).

Both approaches have strong and weak points. SQLite is quite flexible as I recall.

[dl1jbe]

A simple flat file format is easy to work with in any text editor but it is fragile in the sense that great care needs to be take to assure its integrity throughout the program. Using an SQLite database may protect against this somewhat though it is not as strict (as I recall) enforcing data types as Mariadb or Postgresql, for example,

In case of SQLITE that enforcement can be done in the software using the database. You can add authorization functions to verify nearly any aspect of the data handling. It would only be a problem if someone changes the database with a general database editor.

On the other hand, it may be easier to recover a corrupted flat text file than an SQLite db from a corrupted SD card, etc. (often times in those cases nothing is recoverable).

Databases can be exported to a flat text file, maybe that could be used as a good backup.

There is another point you mentioned before:

TLF has a very stable and reliable storage system. As far as we know crashes, power outage and similar events have not led to a loss of contest contacts or log file integrity. If a switch to a database will give us even reliability has to be proven.

Anyway I am pro the use of a database. It would simplify quite some things

[ha5se]

Let me contribute some more considerations:

1. Since the old file format is not truncating trailing blanks, thus only containing fixed length=73 records, thus it could be a backward-compatible trivial few-liner change to accept both formats, and expanding short records by read.
2. The comments pro and con database were rather general, independent of the actual requirements in tlf. In a single contest, even a big gun remains below 10.000 QSO. Our records are relatively short. From the point of database, we only have three columns: callsign, band, and the rest. We only fire simple "is-equal" searches for callsign or callsign+band. Considering this, even the smallest database software is IMHO obvious overkill.
3. Considering that at least 95% are repeated searches on "QSO-before" callsigns for bandmap, and considering some field-day stations using low power-consumption (thus slow) laptops, even if we ever would want improve search,
   • locally implemented simple sorted band+callsign list (~indexing) and simple binary search would obviously outperform any database software, due to the db sw being much more general, much more complex, and designed for much larger data
   • "maintenance" operations that are trivial file processing now, like "list contests", rename previous year, backup, or delete "trial-contests" that I was only playing with, export, import, etc., all those should additionally be implemented for database usage
   • without database, improved search could be gradually implemented, first only the index list, then e.g. selectable search by developer config option "USE_INDEXED_SEARCH", etc. This would enlighten the testing, the introducing, and even later bug reporting. Databases, on the other hand, could only be introduced with much more effort.
4. Converting to database would also need additional effort to implement and test package dependency for each distro involved, not forgetting documenting HOWTO INSTALL, etc. including referenced webpages ...

All of the considerations so far, including mine above, are from technical point of view. I know weighting is of personal taste, still for me, there are much more "cons" than "pros" introducing database in TLF, too much work, small positive, partly negative impact. However, there are some other, call it "social" aspects as well:

1. For DB experts: "if all you have is a hammer, everything looks like a nail..." ;-)
2. This is open source, and many existing and possible contributors are unfamiliar with databases. Even in my workplace, large company all around databases, professional DB administrators reject working with any other database implementation except of their preferred one.
3. IMHO, for each open-source project to survive, it is very important to maintain enough active contributors. Database would be one more complex element and would make it more difficult for inexperienced contributors to join, eventually even to code and test the simplest improvements idea... (believe me, it is hard enough to join right now, I am a recently joined rookie myself ;-) ) ...

[dl1jbe]

Valuable thoughts which should be remembered in the final decision.

Let me just answer to some of them and give some more background there.

1. Since the old file format is not truncating trailing blanks, thus only containing fixed length=73 records, thus it could be a backward-compatible trivial few-liner change to accept both formats, and expanding short records by read.

Partially possible. It was done before as we added the working frequency to the log (last 8 characters). More difficult to apply if you quite often have to modify the scheme to adapt to changing requirements.

2. The comments pro and con database were rather general, independent of the actual requirements in tlf. In a single contest, even a big gun remains below 10.000 QSO. Our records are relatively short. From the point of database, we only have three columns: callsign, band, and the rest. We only fire simple "is-equal" searches for callsign or callsign+band. Considering this, even the smallest database software is IMHO obvious overkill.

It is not quite as easy. Besides the things you named here TLF maintains internal quite a lot of statistical data - which countries, prefixes, multipliers and stations worked on which bands. Which station was first to make a multi, how to handle if that QSO is corrected later. How many point the station has gotten in last run and some more....

These tables needs to be built each time you start TLF by reading the logfile and they need to be manipulated when inserting, changing or deleting QSOs. It is doable as TLF has already shown, but a lot of work.

The downside of the actual scheme - it is not easy to extend. And looking to newer contest with quite a lot of 'creative' scoring rules there is a need for a more flexible and extensible way of storing relevant information's. And that is just what databases are good for.

Lets have an exampe: As soon as you work a new multi it got an entry in the last columns of the logline. That is used in next start to count the worked multis. What happens if you change or delete just that line - you have to look through the whole file for another 'first' on that multi and mark it. On a database just ask which of all QSO's with that multi were the oldest. Done.

   * "maintenance" operations that are trivial file processing now, like "list contests", rename previous year, backup, or delete "trial-contests" that I was only playing with, export, import, etc., all those should additionally be implemented for database usage

If we use SQLITE and keep one database file per contest, that will do not change much.

1. For DB experts:  "_if all you have is a hammer, everything looks like a nail..._"  ;-)

Good that we have no DB experts here :-) (at least as far as I know).

As said above - most other points are valid concerns.

[ha5se]

Thank you for your to-the-point valuable notes. Still I'd like to argue ;-) ... (well, no wonder, regretting and admitting being Ariel [=Widder] ;-) )

Yes, I knew that later data changes would need re-evaluation.

However, I am afraid it is still more complex. I did not know that multipliers are stored in the flat file now, I assumed that at restart it is completely re-evaluated anyway. In QSO files, I would never think of storing "computed/extracted" data (that I can re-calculate any time later), in addition to "original" data. Waste of space, since after restart you cannot count on that data anyway, since both flat files and also db files could be modified using native (non-tlf) utilities. But even without QSO-file modifications, just assume I only refresh the country list, or select another or updated contest rules during the contest: both may result in another set of actual multipliers, even after a simple :CFG ... This means that for re-start and :CFG, complete re-scan and re-scoring must be implemented anyway. IMHO it is simpler to use this code also for later data change or delete, instead of implementing additional maintenance of extracted tables... Assume change or delete a line that is actually multiplier, and you make a db scan -- if there you find another one then replace the pointer, otherwise must remove the multiplier from the list -- IMHO unnecessary code path considering how seldom you actually need it...

IMHO, updating existing QSO during contest is rather seldom: sometimes correcting exchange (zone or section) when I hear him much better, or very seldom when I am sure it is the same station whose callsign I am sure I want to correct now in an old QSO ... I think I should never delete existing QSO... If I am unsure whether it was done, still I'd rather make a dup with him, but leaving in place the old QSO as well. Additionally, in case of serial number exchanges, this would leave a hole in my log....

I know that some functions could be made with less lines when calling external products in place of own code. However, it would need to replace existing code, need test, and in end effect would result in slower execution and generally larger footprint, while advertising the opposite...

[airween]

Databases can be exported to a flat text file, maybe that could be used as a good backup.

my idea is turn the direction and export flat text file to database (eg. after every contest).

The next step is when user can chose his/her preferred storage: plain text file (robust, as you wrote), or SQL.

These are on my (imagined) TODO list. With help of this aggregated data the user can makes many statistics (DXCC and others...)

Note: I made a tlfstat "extension" a few years ago which isn't published, the idea came from here, and after a quick discussion with @df7cb.

[N0NB]

Many years ago I came up with a schema of sorts:

           CREATE TABLE IF NOT EXISTS Log
           (Id INTEGER PRIMARY KEY ASC,
           DateTime_On INTEGER NOT NULL,
           DateTime_Off INTEGER NOT NULL,
           TX_Frequency INTEGER NOT NULL,
           RX_Frequency INTEGER,
           TX_Mode TEXT NOT NULL,
           RX_Mode TEXT,
           Station_Callsign TEXT NOT NULL,
           Worked_Callsign TEXT NOT NULL,
           RST_Sent TEXT,
           RST_Received TEXT,
           Exch1_Sent TEXT,
           Exch2_Sent TEXT,
           Exch3_Sent TEXT,
           Exch4_Sent TEXT,
           Exch5_Sent TEXT,
           Exch1_Received TEXT,
           Exch2_Received TEXT,
           Exch3_Received TEXT,
           Exch4_Received TEXT,
           Exch5_Received TEXT,
           Op_Callsign TEXT,
           Op_Name TEXT,
           Radio_Number TEXT,
           Position_ID TEXT,
           Event_Name TEXT,
           VFO TEXT,
           Note,
           SHA1SUM
           );

It is undoubtedly incomplete. A number of fields would be unused in a typical contest and there are probably a few that would need to be added that I did not anticipate, i.e. general QSO logging where more info from the other op would be logged, name, QTH, etc. It's possible that not so many exchange fields would be necessary but I was thinking of ARRL November Sweepstakes. My thought was to be able to analyze the log after the event based on any field of interest.

[dl1jbe]

Something like that would be needed for database use.

       CREATE TABLE IF NOT EXISTS Log
       (Id INTEGER PRIMARY KEY ASC,

.... Worked_Callsign TEXT NOT NULL,

Better have a second table with all information about these operator and just add a reference to it here. There can all the information for that OM/YL like name, QTH, country and prefix, initial exchange, ... They are the same for any QSO with him.

[dl1jbe]

Yes, I knew that later data changes would need re-evaluation.

However, I am afraid it is still more complex. I did not know that multipliers are stored in the flat file now, I assumed that at restart it is completely re-evaluated anyway. In QSO files, I would never think of storing "computed/extracted" data (that I can re-calculate any time later), in addition to "original" data.

If we start anew that is an option. The actual situation is so that we still use a slightly extended TRLog format. And please do remember in times before cabrillo TRLog was an accepted format to sent in your contest results or it could be printed on paper and then sent in.

[ha5se]

The next step is when user can chose his/her preferred storage: plain text file (robust, as you wrote), or SQL.

A very clever and elegant suggestion, I like it. And to keep footprint small, how about ./configure --backend={flatfile|sql} To maintain code readibility, flat and sql code could even be split into completely independent files...

[zcsahok]

TL;DR

• we could use a subset of ADIF for log storage
• internally QSO data could be put into a DB table allowing contest-specific plugins to evaluate it

IMO we shall make a distinction between 1) the internally used data structures 2) and the file format used to store (but not to process) QSO data.

For internal data management we a free to use whatever technology fits. Currently for this we use several historically evolved lists and they serve their purpose quite well despite using plain linear searches. The big disadvantage of this approach is the lack of flexibility: we can't just add another "tricky" contest. This we could overcome by inserting the raw QSOs into a DB (SQLIte) table and devise a plugin mechanism so that a contest-plugin could implement methods like is_multi() (see #235) using this data. Any additional internal data (multis, whatever) a plugin needs it could simply store in the same DB in separate tables.

The external data ("backup" or journal; this is our .log file) is our master data. From the functional perspective it shall be 1) extensible (this is what we lack now) 2) allow fast appending (adding a QSO), and 3) quickly parsable to restore the internal state.

At 2 and 3 we OK now.

From user perspective (IMO) it shall be relatively easily viewable and editable. (use case: a non-tlf-pro op wakes me up in the middle of the night during a multi-op FD contest that 20 Qs ago he made a typo)

Both could be rather easily be covered by the ADIF format: it is well known and can be easily extended. If one keeps 1 QSO/line then it is quite easy to view/edit. Of course it is pretty verbose, we can assume a factor 2 increase in file size. But that's still not a big issue. And it is relatively straightforward to parse (even easier if we don't aim for importing arbitrary ADIFs). On startup it's easy to decide whether we have an ADIF file or the current format.

(Note that using a non-text external format and still having :EDIt available means implementing the export-to-text and import-from-text functionality. So we back at text again. Then what's the point having our external data in non-text?)

[df7cb]

Hi,

as already mentioned by @airween I have my logbook in a PostgreSQL database. But for contesting, I love the simplicity of the tlf text file format: if anything goes wrong I just edit the file [*] and am back on track a few seconds later.

If you move this to SQL, making any fixes will require the user to write SQL commands, or you have to write a lot of code to transform any edits in the tlf ncurses interface into these SQL commands. Except for the cases where the user does some unexpected edits and there is no SQL command for that yet and you need a generic fallback editor.

Don't go down that rabbit hole please. It's easy enough to import a cabrillo log into a database after the contest.

[*] I enter the editor by pressing 6 times, and then in "less", press "v". There's probably a better keybinding for it, but this works for me.

[airween]

And to keep footprint small, how about ./configure --backend={flatfile|sql}

I should prefer

./configure --backend=tlf,sqllite,mysql,pgsql

because (imho) they aren't mutually exclusive, but if you don't want to use any of them, you can leave it from list.

Btw, in first step, I just think about flat -> sql direction, and keep the text file as primary backend.

[airween]

Hi Christoph,

thanks for your note :),

as already mentioned by @airween I have my logbook in a PostgreSQL database. But for contesting, I love the simplicity of the tlf text file format: if anything goes wrong I just edit the file [*] and am back on track a few seconds later.

This is a very important thing! Adding the features (like post editing QSO's) for any supported backends should be painfully.

Don't go down that rabbit hole please. It's easy enough to import a cabrillo log into a database after the contest.

it's easy if you just focusing the QSO's itself.

If you want to go away and (for example) wants to make any stat about your QSO's (DXCC per modes/bands/...) it would be a bit harder, because you have to implement the function which expands the callsign to DXCC - Tlf already knows that. Same issue if you want (just for example) to see the prefixes.

[ha5se]

./configure --backend=tlf,sqllite,mysql,pgsql Once more, I absolutely agree. Other applications may already be bound to other database(s), e.g. cqrlog to mariadb, and it is not very nice to be forced to install a different db for each app. I thought on this myself -- however, I felt it would be too much from me for today... ;-)

[airween]

For internal data management we a free to use whatever technology fits.

The external data ("backup" or journal; this is our .log file) is our master data.

I should change these definitions. I'd keep the text as internal and any new backed as external.

Don't forget the "old" users, who don't want to use any SQL-like-magic, just want to edit the log with a simply editor. Therefore the mentioned functions should be export-from-text and import-from-backend.

Then what's the point having our external data in non-text?

In first step we can use it to archive QSO's, make master callsign database, and so on. Later we can add a new functionality which store (or update) all QSO's in external db when a new one had created or updated in internal. But it would be optionally.

[airween]

Something like that would be needed for database use.

       CREATE TABLE IF NOT EXISTS Log
       (Id INTEGER PRIMARY KEY ASC,

....

       Worked_Callsign TEXT NOT NULL,

Better have a second table with all information about these operator and just add a reference to it here. There can all the information for that OM/YL like name, QTH, country and prefix, initial exchange, ... They are the same for any QSO with him.

I should store more data about QSO, which Tlf produces, eg. DXCC, prefix, and similar...

Once I mailed with Jim AD1C, and asked him about historical changes of DXCC. He mentioned a good example, TO5M:

TO5M,0,FM,05/03/1993,28/03/1994 TO5M,0,FP,27/10/1995,01/11/1995 TO5M,0,FR,23/10/2004,01/11/2004

So TO5M counted as FM from 1993 to 1994, FP in 1995 and FR in 2004. That's why important to store the current info. There isn't any aggregated historical public database. (The Clublog stores this but you need an API key to access it.)

But this raises a new problem: how can a user update the cty.dat as easily?

I can imagine a new cli option, eg. tlf -u downloads and install cty.dat, and/or tlf -m catches the master callsign database... But this is an another topic :).

[zcsahok]

how can a user update the cty.dat as easily?

(in the working (current) dir: rm cty.dat ; wget http://www.country-files.com/cty/cty.dat)

[airween]

(in the working (current) dir: rm cty.dat ; wget http://www.country-files.com/cty/cty.dat)

Sure, but it would be so elegant if the logger can do that :).

[N0NB]

So TO5M counted as FM from 1993 to 1994, FP in 1995 and FR in 2004. That's why important to store the current info. There isn't any aggregated historical public database. (The Clublog stores this but you need an API key to access it.)

I've not worked a DX 'test in a long time, but for all contests I keep them in their own directory. I keep the mult file(s), master.scp and the cty.dat from that time of the event in that directory along with the log. In theory it would be possible to recreate the data conditions at the time of the event. One could even go so far as to build Tlf from that time as well!

[airween]

I've not worked a DX 'test in a long time, but for all contests I keep them in their own directory.

Few days / a week ago Thomas wrote there are two kind of users: one follows this way as you mentioned here (like me :)), and another one who creates a separated rule directory, and collects logs in same working directory.

Both of them are valid, but only the first way (preserving the cty.dat files) can reproduce the previous states.

I just want to say if Tlf can produces the necessary data in runtime, we can store that information in the backend storage, and don't need to keep the old files, it can work for all solutions above.

[m5evt]

I set myself a small project over Christmas to try and do some mapping and provide some stats from tlf. Things like rates for last 10 Qs, last 100 Qs, On times/off times, graphing these rates etc. Doing all this in raw c code just seemed counterproductive. It seemed most sensible to do this in a database. I had never really done any db work before and was able to easily lash together queries to achieve what I wanted. There are so many tutorials on the net.

I was also mapping cqww zones, states worked etc. I did this all in Python and using MariaDB (and matplotlib for the graphs). Some of the commentary around this is in #203. sqlite vs MySQL seems a hot topic that gets debated by even the experts. I read enough to convince me that MySQL (and MariaDB) seemed best to invest some time in due to potential for multiple clients writing at the same time. Granted, sqlite seems much simpler to setup.

Right now it sits as a bit of a mess of Python code but I do hope to tidy it up (in slow time) to share it for others.

It do feel that if tlf used a database, more "add on" applications such as this could be created without affecting/annoying tlf devs. I would of thought the network code would all much much simpler with a db?

Regardless of where tlf goes in future, thanks to all the tlf devs. It is great to see tlf dev active. I really do like the (keyboard driven) interface.