Clocks does not sync after long periods of flat out signal

[udoklein]

Logfiles found here: https://github.com/nameoftherose/dcf77-leonardo_support/tree/master/logfiles

indicate that the clock has major issues of syncing after prolonged periods of signal fading. The issue is caused as follows: the decoder chain processes data no matter what the signal is. This is in order to get rid of noise. However faded signals are not noise. Thus the decoder chain assumes this is a noisy signal which consists basically of 0s. In particular the year decoder will then collect this and decode (the valid result) 00. One a (possibly noisy) kicks in again the year decoder may take more than an our to get rid of the 00 data. If the signal is not available for such a long period of time this prevents the clock from acquiring any lock at all.

[udoklein]

This is tricky.

There are several sitiuations to consider:

1) Clean signal 2) Signal with huge noise on top 3) Signal that fades more than 90% 4 Signal that fades and has lots of noise

Best strategies (with limited memory):

1) Naive decoder 2) My current library 3) Naive decoder + consistency check, e.g. if it decodes consistently 3 times in a row the time should be correct. 4) This is the real challenge. 4a) Add a fading detector to my library and react accordingly. However how to detect the presence of the signal in a big noise floor? Usually this means to narrow down the bandwidth. This tends to make fading detection harder. --> will investigate in this direction 4b) Assume a lower noise floor. That is lower the bandwidth. --> will also investigate in this direction 4c) Add several detectors with different bandwidth and see which one catches a reasonable signal. This is basically creating several clock instances with different bandwidth and will increase memory consumption a lot. --> not feasible for AVR but could be made to work for ARM. --> might investigate in this direction. 4d) Full scall FFT approach --> not enough memory on microcontroller --> not feasible.

[udoklein]

4c is definitely only possible for ARM / Arduino Due because the library already consumes a lot of CPU cycles. So I will first try to find a solution that would be possible for the others as well. However there is already a 8 MHz version out that is close to limits of what is possible with AVR.

[udoklein]

Maybe 4a could be a feasible solution.

The basic fading detector would be a phase lock with a higher bandwidth. E.g. filter constant such that it releases its lock after the signal fades for more than 2 minutes. This in turn would purge the other filter stages immediately. This would degrade the overall noise tolerance but improve performance with regards to fading a lot.

[udoklein]

Looking into the log files again it appears that the clock tries to decode even thoug the decoder's phase lock is abmysal. Maybe the best way is to just change the threshold of what is considered a reasonable phase lock.

Typical signal during "good" phases looks like this:

    62724, +---------+---------+--------32---------+---------+----2X434+---------+---------4XX1------+---------
    62725, +---------+------2XXX49XXXXXXXX3--------+----3683-+---------+----3X542X8X5------+---------+--3XXX---
    62726, +--------1XXX3------3X4-------+--1XX5---+--15-----+---------+----4X2--+--6X3----+---------+---------
    62727, +35-------+-----7X3-+-----1XXX6---------+---------+--2XX6---+---------+---------+---------+---------
    62728, +---------+--1X8----+-43XX7XX4+---------+278XXXXX4+-------589XX67--1XXX4--------+---------1693----85
    62729, +-------74+----1XXX5+---------XXX-8XX4--+---------+---------+----28X7-+-----55--+-----4482X5--------
    62730, +---------+---------+---5-----XX19X7-8XXXX6-X1----X5----5X4-+---------96--------+---5X9-X7X9--------
    62731, 6X4---875-+-------46+-98XXX5882-------6XX-9-------+---------7XX2------+-----XX1-+-----61--+---------
    62732, +---6---78+8XXX8----+-3189-4XX3----238-5+-----6XXXX8-----497+---------+---------+---------+-----18--
    62733, +---------+---------+-----47XXXXX86XX8XXXXXX99-2--+-------71+---------+---------+---------+---2XXX8-
    62734, +---------+2---4----3X16X36XX34XX-6XX83X9XXXXXXX83+---------+---------+----141--+---------+---------
    62735, +---------+-14-----2XX1-----541X211-----+-1-24----2X3-------+-----386X3------3XXX3-4------+---------
    62736, +---------+--71-38X68X6XXX3-5XXXXXXXXXXXXXX6------+---------+---------+--3XXXXXX477X7X2---+---------
    62737, +---------+--6XXX-1XXXX64-262XXX3-4XXX59XX7656----+-------4-3X75------+---------+---------2X4-----3-
    62738, +---------+----89---+------36XXXXXXXXXXX1---------+---------+---1X56--+--------13---------+---------
    62739, +---------+---------+-5-------254------3+--792----+---------+------6--+---------+---------+---------
    62740, +-------1X555-------+---1XX8--+---------+---88----+------X6-+X7X5--7--+----95---+---------+---------
    62741, +---------+---6-----+---------+-5-9X6---+--------95---------+---------+---------+------97XXX55XX5---
    62742, +3--------+---------+96XXXXX98+96-------+---------+--3XXX9X5+---------+---------+--X98----+---------
    62743, +7---6XX9X6---------+--41XXX5-+974X7----+---------+---------+---------+9X2-8XXXXX8--------+---------
    62744, +---------+---------+--9----1X63XXXXXXX8+--------X5---------+---------+---------+---------+---------
    62745, +------XXX6---------+8-------XXXXXX7-XXX5---------+---------+------XXX6-X7------+---------+---------
    62746, +-----1XX8+---------5X671XXXXXX-1XXXXXXXXX97------+---------+---------+---5--967XX6-------+---------
    62747, +---------+----5XXXX5----6----+-8XXXXXXX6866-75---+-------2-+4---8----+---------+95-------+--------4
    62748, 987XX94---+---------+-1XXXX8--+--------4XXX7------+---------+--7X948--+---------+---------+------953
    62749, 61----1X6-+---------+XX86X68--+---6XX949796XX6----+---------+---------+---------+---------+---------
    62750, +---------+--------698--7969-9XX6-8XXXX7+---6X881X6-467-----+---------+--------X89--------+----55---
    62751, +---------+---3----3XX7-------+---------+--------46---------+----3----+----7XX5-+---------+---34----
    62752, +---------+4X379-3X9+----8XX8X+XXX8XXXXX+---------+---------+---------1------4--+----9----+---------
    62753, +---------+---------+--31---398X19XXXXXX99-XXX----+---------+------8XX7----4XXXXXX88X8----+---------
    62754, +--678XXXXX15XXX8---+--------89519XXXXXX1---------+--------59---------+---------+---------+---------
    62755, +---------+-4XXX1---+---XX25XX2----14---+--71-----+---------+42-------+---------+--3253---+---------
    62756, +---------34XXX39715X-27X1----+45XXXXXXX2--72-----+---------+---------+4X2--3XXX4----38---+---------
    62757, +13-------+---------+----6XXXXXXXX4-----+-2X3-----+2XXXX44X23----2----+--1X6X1--+---------+--------4
    62758, XX4-------+---------+4X8XXXXX1+--3X3----+57X3-----+---------+---------+---------+---------+-------4X
    62759, 1---------+--3XXX2--+4X1----2-9X-3XXXXXX4---------+---------+---------+-7XX1----+---------+---------
    62760, +---------+---------+-6XXXXXXXXXX3XXXXXXXXX34--45-+-------97+---------+---------+--25-----+---------
    62761, +---------82-3XX5--1X74-5-----+-1XXX7XXX6-775-----+--3XXX39XXX5-------+---------+-------84+----2XX73
    62762, +--------114-6X2--165----1XXXX6-71XXX4--+---64----+---------+---------+---------+17-------+---------
    62763, +---------+---------+2X554----+--4-8XXXXXXXXXX82XXX5-1XXX4--+---------+--6-4-4X5+---------+-----95--
    62764, +---------+------45XX7X6XXXXX6+---------+---------+---------+-7X8-----+---------7-7-------+---------
    62765, +-----5-3-+56-------+-8XXXX9XXX8--------+---------+-------68+---------+---------+---------+------9XX
    62766, XXX7------+--5-----9X8X5X--3X-+---------+---82----+---------+---------+---4XXXX-+---2---6-+---------
    62767, +---677-5-+-2X98----+-9-748X98+-4---66X8968------53---------+---------+---------61--------+7X981----
    62768, +---------+---------8891----88+-616XX8--+---------+------2--+---------+------589+4XXXXXXX9+-29---29X
    62769, X7X9------+--------38XX25XXXXX47--3XX3--4---825---+-3X------+-----5---+--X--39--+---------+---------
    62770, +---------+231-----55XXXXX98--+----2----+-2--5XX82+4--------+---------+----31XX9+---------+----5X---
    62771, +---5X7XXXXXXXXXXX9-+----97XXXXX1-5XXXXX9XXX9X----+---------+---------+---------+---53----+35X5-----
    62772, +---------+---------5X74------+----1XX--+---------+----2XX-41334-X1---581--93---+---------+---------
    62773, +---------+----43--X7XXXXXXX2-+-216X4XXX44--------+---------+-------2X382----73-+---------+--------8
    62774, XX3-------+-------4XXXX5------+-----1XX54XX8X5-956+---------+-36------+---16----+---------+---------
    62775, +-------17+--------7771X487--4+-1X-XX5--+--418----+------65-+-3--95--5+------1--+---------+---------
    62776, +------84-+---------+-1X6-8XXXXXX27XXX6-+------9X9+---------+---------+---------+---------+---------
    62777, +---------+---------6X7-------+---------+---------+-----6X8-+---------+---5----511-------8XXX8----6X
    62778, X9---4-67X95--------+------7XXXXX1XXXXXXXX1-------+---------+--6XX1---+---------+---------+---------
    62779, 4XX7------+---------+5XXXXXXXXXXX5XXXXXXX72-------+---58X5--+--59--52-3----7XXX3X---------+-473-----
    62780, +---------+---------3X3685937XX--7XXXXXX9X2-------+--2X1----+---------+------2X1+--5XX----+---------
    62781, +---------+---------+4253--64-+----5XXXXXX3----6X8X1--------+---------+--4XXX4--9X3-------+---------
    62782, +---------+---------+--5XX4243X284XXXXXXXXX8X4638XX3-1X322--+---------+-1XX2----+---2XX2--+-----81-3
    62783, XX4-------+---1XXXXXX6X6X6X6X2+-1XXXXXXX5------3521---------+--2X44---+---------+---------+---------
    62784, +---------+-----6XXXX761X96-5-12--------+---------+---------+---------+---------+--26-53--2514------
    62785, +3845-----+-----63--XXXXX6XXXX4---------+--55-----+---14-42-1-93------+---------+---------+---------
    62786, +---9X7577X5---5XXX6XXX561X342X91-X55XXXX52-------+---------+---------+---------+---------+---------
    62787, +-4-------+----2----+------2--+---------22XX7-9X7-+---------+-----95--+---------+---------+576-5----
    62788, +-----X8XX89X6XX8-XXXX8-3X8-X6+-X9-1XXXX8---------+------9-9X64-------+---------+---------85--------
    62789, +---------XXX8--4---+-----278-+7-7XXX7-3XXX8------+---------+---------+----39757+---------+---------
    62790, +-----9-8-+---------4X46X8X---+---------+-6XXXX---+-----888XX7--------+---------+---------+------5--
    62791, +---------+-------4XXXX9X-7171+--5XXXXXXX-26------+---------+---------4XXXXX9---+---------+---------
    62792, +-4-3543695X9-----4-+-2XX1376XXXXX4XXXXXX9--------+---------+---------+---------+---------+---------
    62793, +-8XX2----+-------5XXX7XX2----+---4XXXXX+-4XX2----+------6XX+---------+----6X35XXX71------+---------
    62794, 5XXXXX----+4XX------+-574X----+----27XXXXXXXXXX323+36X-3-4XX8---------+---------+---3X83--+---------
    62795, +---------+---------4XX42347--+---4XX2--+---79-28X34--------+---------+--92-----+66X6-----+---------
    62796, +---------+---1X4-3X2-3XXX4---+-4X2XXXXXXXX35X1---+---------+---------+--48X8X423X55------+------76X
    62797, XXX434----+---------+----1XXXX3----8XX6-+-2435----+---------+---------+---------+---------+----XXXXX
    62798, XXXX31X5--+--------X3635--1525+----1XX3-+-3X3-----+-----XXX72----5----+---------+---------+--------6
    62799, XX4-3XX67X3--------5XX8X9XXXXXXX2----9XX3----1--285--X4-----+---------+----15---+---------+---------
    62800, 1XXXXXX3--3---------+---361X5-+---------+93--2----+---2X4--2XX4-------+--------XXX4-------+---------
    62801, +---------+--------2XXXXX63XXX6X1-------+-46------+---------+---3XX6--+----X4---+---------+--2X-----
    62802, +---------+----XXXX44XXXXX8XX73----1XXXXXX4-------+--------9X319X5----+---------+---------+--------9
    62803, 1---------+------9XXXXX6--76--+---------+6-5------+---------+-----36--+---------+---------+--63-----
    62804, +---------+------72XX6-----9XXX6--58XXXX665X4-----+---------+-----2X3-+---------7XX417----+-2X5----6
    62805, 4------1X7+---------+-262XX6--2-----22-XX7---4----+---------+-----3XX9X7--------+X4--5----+--------1
    62806, 56--------+------9XXXXXX4-7XXX9--7XXXX6-+-26-3----+---------+---3XXX86+---------+---------+-------1X
    62807, 56--------+-------7XXXXXXXXXXXXX28XXXXXXXXXXX6----+---------+------XX7+---------+----41X65+--------8
    62808, 6---------+---X76XXXX7--2XXX94+------5--+---------+---------+----X57--+---------+---------+4X53-----
    62809, +---------+-----4-79561XXX8X5-+---------+--5-----75-3--65---+---------+---------+---------+74-------
    62810, +---------+---------+------XXX4---2-----9XXXX846--+---------+----862--+25-----8X86--------+----XXXXX
    62811, XXX6------+---------+5271X4---+---------1X75------+---------+--------6465-------+---------+---1XX3--
    62812, 1X5--4----+---------2X878X3---+---------+32X9XX85X5---------+--2------+---------+---------+--------5
    62813, 4-----9X8-+---------2XX6-6XXXX5----7XXXXX7--------+---------+---------2X7--1X2-3+7--------+-------4-
    62814, +----9X2--+---------+---------+55---26--27-4XX4---+---------+------X8-+---------+---------+--9XX4---
    62815, +---------+----65-X4287XXX5---+---------+6------1XX7--------+---------+---------+---------+--------5
    62816, XXXX6---16+---------+-1XXXXXX83--6XX3XXX58XX4-----+---------+---------+-5-------+---------+-64X9---9
    62817, 76--------+-------3-+XXXXXX8--+6--------+---------+--66-----+---------+---------+---------+---------
    62818, +--------2+---------+--6---87-3---XXX6--+--7XX77--+---------+---------+---------+---------+8XX8-----
    62819, +7--------+--------5+9XX-8XXXXXXXX-7X7--+--4-6----+---------+---------+---2-----+4XX72---89---------
    62820, +---------+---------+--37XXX7699-5XXXXXXX8XX9-----+-----61--+---------+-------6X1---2--3151---------
    62821, +---5X8---+--------3XXX7XXXX-7XXXX179---+---------+---------+---------+---------+----69---+------61-
    62822, +---------+--------6XXXXXXXXXXX-------XX86XXXX2---+---------+---------+---------+-31X1----+-3-------
    62823, 9X2-------+--5X19XX2+---------+--------3XX42XX32X23XX3------+---------+--2X1----+--4X1----+2XXXXX2--
    62824, +---------+---------8382------+-----3XX2+---------+--------1X4---1X3--+---------+---------+---------
    62825, 1---------+-----14X3+----3XXX4+---------+---------+183------+---------+-8XXXX6X7+---------2XX4------
    62826, +---------+-----55XXXXXXXXX5XXXX6-------+-9XXX8374+---------+---------+-8X7-----+---------+------7XX
    62827, X5--------+5------86X57-X93---+---44----+-73------+---------+---------+---------+-----929-6X86------
    62828, +-438-----+---------+-----586-+---------+---4-----+---------+-83--8991+-87---28XX99XXXX1--+---------
    62829, 7XX6------65--------786XX9----+---------+-7XXXXXX9+-39------+---------+5X9------+---------+--55XX2--
    62830, +---------+------38XX2--5X----691-------+---286---+---------+---------+----4XX2-+-----62--+------97X
    62831, 3--------24XXXXXXXXXX2--53----+---------+---6XX9--+---------3X7731----+---------+-7XXX----+---------
    62832, +---------+-------4XXXX4----7XX1-183----4XXX1-----+---------+---------+---------+---------+---------
    62833, +-54--63--+--------37364-29743+----XXXXXX2--2X1-61+--4XX1---+--------6XX1-3XX2--+---------+---------
    62834, +---------+-3X75-95-+------2--+-----1XXX633XXX1---+4XX---4--+---------+---------+---6X1---+-5X------
    62835, +---------+---------5XX259X869XXXX1-5X9X8X5XXXX3--+------9--+---------+---------+-65------+------42-
    62836, +---------+-------2XXXXXXXX9--+---------+-4XX358X3+---------+-------41+5XXXXXX1-+---------+---------
    62837, +--2----53+---------+-236XX87X1-6XX16X3-+---------+---------+---------+39X1-----+---------+---------
    62838, +---------+-------3XXX5X73--6XX9-4XX----+---------+---------+---------+---------+---------2X12------
    62839, +---------+---------2XX3----22+-----2---+-------6XX9--------+---------+--------4+---------+4-5X3----
    62840, +-------52+------XXXX9X2----2XX9-5XX--6XX2--------+---------+---------+27XXX8X2951--511151+----3----
    62841, +---------+--65X6---518XXXX2--+---------+---------+--7-----7X1--------+-----8XX8+---71--58+---------
    62842, +-------6XX9----7XX8X7--------+---------+-9XXX7---+---------+-9XX-----+---------+-98------+---------
    62843, +---------+---------6XX87XXXX7+-97------+-----8---+--------89---------+---------+-----8---1X8--3----
    62844, +---------+------5688XXXXX9---97------85+---------+---------+---------+-------4XX-----5-7-+---------
    62845, +---989XX-+-----9XXXXXXXX7----+---------+-77------+--------7+---------+---------+---------+---------
    62846, +-88--7847+-----5XX8X7--------+---------+-6-------+---------+-----5---+---------+---------+---------
    62847, +---------+---------25279-----+---------4894X17XX-+---------+---------6X--3XX9--6---------+-----28--
    62848, +----5XXXXX979-4----+-7XXXX18X168XX-----+---------+------4--+---------+---------+---------+---------
    62849, +---------+---------5XXX4XXXX95X-6XXXX3-31----1XX9+---------+-6X9XXXXX+---------+---------+---------
    62850, +---------+---------+---7XXXXXXX---5XXX3+23347835X+---------5X--------+---------+---------+--12XX5--
    62851, +---------+-----2X2-5XXXXXXXXX4X4XXXXX5-+5XXXXX5--+--------X44----35--+---------+---------+---------
    62852, +-------5XXXXX4-2--2XXXX6X2---+---------4XXXX6---73----14---7XX5------+-2-------+---------+---------
    62853, +---------+--7378XXXXXXX79XX5-+--------X2-----17-6+----8575-+---2X3---+---------+---------+------12-
    62854, +---------+--X4-----+----2----+---------+---------+---------+---------+-------5-+---------+--36---9X
    62855, 5---1X52--+---------17--------+---------+---1X7826+---------+---------+---------+---------+-------5-
    62856, +---------1X8-4-----+---------+72XXXXXXXXX64------+------8X1+---------+---------+---------+---------
    62857, +---------+--------X78XXXXX7--+---------5X4-------+---------+---------+---------+---------+--XX56---
    62858, +-----55--97--1XXXXXXXXX59X7--+---5--42X7XX8------+---------+--886----+---------27--1XXX1-+---------
    62859, +--7-7X7--+XXX5-----+----84---+---------+-1X53----+---------+---------+---------+---------+X9XXXXX2-
    62860, +---------+--14-3XXXX6-4----56+---------XX972XX5--+---------+------5--+---------+-------3X3----4X727
    62861, 3---------+----9XXXX3----5----+---------+----955--+---------+---------+---62----+---------+---------
    62862, +---1414--+--46X284-21--------+----12---+3477XX4--+---------+---------+---------+---------+--8X4----
    62863, +---------+---------+-434XX62-+-------2X1-----5XX44X7X1-----+---------+X1-------+---------+---------
    62864, +---31----+-------4XXXX---5XX7XXXX-8XXXX+---------+--------XX2----27--+---------+---43--631XX1------
    62865, +---------+---------2X5XX4----+------8XXX---------+---------+---------+---------+---23----+---------
    62866, +---------+-------2X1-5X3XXXXXXX5XXX171-+---------+38XX4----+---------+----45XXXXX53------+---------
    62867, +---------+-----4XXXXXXXXXXX6XXXXXXXX3-592--7253--+---------+---6XX7--+5X51X1---+---------+--6X8--5X
    62868, 9--3XXX2--+-------5XXXXXXXX1--+---------+---------+-----4XX6+---------+---------+------XX2+-------7X
    62869, 1---------+-5X2-----+-2-------+-1XX1----+---------+---------+---------+---------+---------+---------
    62870, +---------+----7XXXXXXXXXXXXXXX9--------+---------+---------+---6XXX--+2--------+---------+---------
    62871, +---6XX3--+-------5XXXX9------+---4X1---+---------+-----4X1-+---------+----5XX-51---------+-------1X
    62872, 1---------+-------4XXXX2----47XX4XXXXXXXX1--------+--263----+---------1XX3------+-----42-1+---------
    62873, +---------1X3XXXX3--+-3X78XXX4+85XXX633XXX1---13--+-------4X71--------+---------43--------+---------
    62874, +4571--4--+---------+19XXXX945+------1XXX5-164----+---------+---------+--------83---------+---------
    62875, +-------13+----32-3XXX8XXXXXX5+--13----2XXX7------+---------+---------+---------+-----25--+---------
    62876, +---------+---------+-1XX54X7XXXXX17XXXXX635------+---------+------2--+------X488XXX62----+------93-
    62877, +-26------+--------6X61626----+------XXXX6--------+---------+----5----+---------+-----17--+--XXX4---
    62878, +---------+---------+X4-2XXX4-+---------+---------+---------+---------+---------+--3------+-----9X4-
    62879, +--4------+---------+--------96816XXX6-1XXXX4-----+---------+6-6------+---------+--2------+------65-
    62880, +--7-4----+-----1X51+--9XXXX4-+95--68---+---------+---------+---------+--95-----+---------+---------
    62881, +--------825XX8X89XXXX7247---69X2XXXXX3-188X689X4-+---------+---------+---------+---------+---------
    62882, +---------+---------+-17-XX7176X3--7XXXXX84-------+---------+---------+-------2XX747------+-------36
    62883, +---2XXX3-+-----2---7XXXX726--+--9XXXX4-+--3------+---------+-------16+---2-----+---39X6--+---------
    62884, +---------+-----1X36XXX6------+--14-----+-----2XXXXX34--2XXX43--------+--1X5----+----11XX7+--142-83-
    62885, +---------+--------1551XXX4---+--------5+---------+---------+---------+---------3522------+---------
    62886, +-------1X44XX2-----3XX6883---+-3XX1----+-25------+----94---+---------+---------+---------+---------
    62887, +---------14-1XXX8X8X898X424--+----5X6XXXXXX75----+--44-----+---------+---------+--------22---------

[udoklein]

Here is a bad period

    82919, +---6-----+--------6955-------+---------+---------+---------+---------+---------+-------4-+2--------
    82920, +---------+--4XXX---+-77------+---------+---------+---------+-----613-+---------+---------+---------
    82921, +---------48------68+----5----+---------+---------+-------2X+-----62--+---39---3+-72------+-------29
    82922, +---31----+-68-6XXXXX9--------+---------+---------+---------+---------+---------51--------+---------
    82923, 1---52----+---------+---------+-19------+---------+---------+---------+---------+---------+---------
    82924, +---------+---------+-72592---+---------+---------+---------22--------+---------+-523232--+6--------
    82925, +---------+---------777-------+---------+---------+---------+---------+--5------+---------+---------
    82926, +---------+---------54822-----+---------+-----52--39--------+---------+---28----+---------+---------
    82927, 65--61----+---------+-695152--+---------+-1161----+---------+---------+969------+-------31+---------
    82928, +---------+----7XXXXX1---4----+---------+---31----+---------+---------+---------+-------4-+5--------
    82929, +---------+---5XX861358181----+---------+---------+----478--+---------+-----24--+---61----+---------
    82930, 51--------+------5798-717381--+---------+---------+---------+---------+---------+---------+---16----
    82931, +---------+-----6XX-5181636---+---------+-----61--+---------+-71------+---------+---2-4-1-+---------
    82932, +---------+-------89X---------+---------+---------+---------+---1-----+---------+---------+---------
    82933, +-4-------+---77----4-5-8-----+---------+---------+------4--+---------+---------+---3-----+---------
    82934, +---------+-----6-3-7-X8X4----+---26----+---------+---------+---4-6---+------3--+---------+---------
    82935, +-------4-+--585--96+------4--+---------+---------+---------+-----3---+---------+---------+---------
    82936, +---------+---8XX573+-7-------+---------+---------+---------+---5-----+-------93+-----3---+---------
    82937, +---------+---6X7---+175X2----+---------+---------+---------+---------+---------+----1X2--+----2----
    82938, +---------+----25---+--3X54---+---------+---------+---------+------23-+-----11--+---------+---------
    82939, +---------+-----3XX6X64------2+---------+---------+---------+---------+---------+---------+-2-------
    82940, +---------+---------+---------+-5---9---+---------+---------+---------+---------+---------+---4-5---
    82941, +---------+--------4+---------+-6-6-----+6-8-----87---------+--7------+------2--+--------5+---------
    82942, +--------3+---------+----2---83--77-98--+988------+---------+---------+---------+--6---2--+64-------
    82943, +---------+---------+---------9X1---4---+---------+---------+---------+---------+-------14+---------
    82944, +---------+72-------+------965+3--------+---------+---------35--------+------1-2+---------+---------
    82945, +---------+---------+---------+---------+---------+---------+---------+4--------+---------+---------
    82946, +---------+---------+---------+---------+-32------+---------+---------+---------+---------+---------
    82947, +---------+---------+-----2---68--------+---------+---------+---------+---------+---------+---------
    82948, +---------+---------+---8659XXXX88X7----+---------+-38------+---------+---------+---------+---66-2--
    82949, +-----8-6-+--------2+-----41XXX7--------+---------+---------+-----65--+---------+-------2-+---------
    82950, +---83----+---------+-----6-72X5X7------+---------+---------95--------+---------+---------+---------
    82951, +---------+---------+1X8136---+3-1XX7---+---------+---------+---------+---------+--------14---------
    82952, +-------62+---21----+--277XXXX41XX1-----+---4-----+---------+--154434-22-2X2----+---------+----22---
    82953, +34-------+-----32--+5X5----3-+4X9-52---+--------3X1--879---+4--------+---------+-----1---+-------5-
    82954, +---------+---------+------4X72----8X---+---------+---------+---------+---------+--52--62-+---------
    82955, +--------574---68---+-------281------9948---------+---------+----766--+-148-368-+---------+---------
    82956, +---------+---------+------4--+--------98---------+---------+7-------3+---------+--6---7--+---------
    82957, +---------+---------+----1--18+---------+227------+--26-5---+6--------+--------24----847--5---3--1--
    82958, +---------+---------+--4--5--9XX659-----+1-84-----+---------16-------5+---------+----4-5--+---------
    82959, +---------+---------+-----27--+--9483-2636--------+-------15+---------+---------+---------+---------
    82960, +---------+--6549XXXXXX615--2-1X6XX5-64455--------+---------+-46------+---------+------525+---------
    82961, +---------+------37X2X95------+---------+2-1--24--1X2-------1---------+---------+---------+---------
    82962, +---------+----32XXXX811----23+------X28+---------+---------+-------2X1---------+------2833X1--41--9
    82963, 1-------44+----44XX923-34453--+------31-+---------+---------+-------41+-----69--+---------+---------
    82964, +---------+------X4XX2-3------+---------3---------+-----2X2-+-------31+----8----+---------+--------3
    82965, +---------+--2--68--+---------+---------+---------+---------7---------+374-6X977+---------+-------28
    82966, +---7-----+---78384883----62X71-29X7----+---------+---------+---8698X32-1X9-6---+---------+---------
    82967, +-------3-+---------+----2----+4--------+---------+---------+-------6-+---------+--1X4X---+6-----562
    82968, 3---------+-64XXX5--+---------+------2--+---------+---------+---------+---------+---------+---------
    82969, +---------+---------+---------+-2-------+----366XXX633------+---1-----+--------1X962X34---+---------
    82970, +---------+-64------6---------+---------+---------+--------47---------+---------+---------+---23--83
    82971, 5---------+--X4-3-1792--------+---------+---------+---------+------22-+---------+---------+---------
    82972, +---------+-----12--+---------+---------+---------+---------+---------+---------+---------+---------
    82973, +---------+---------+---------+---------+---------+---------+---------+---------+---------+---------
    82974, +---------+--------2X8X1------+-----1---+---------+---------2---------+-6---5---+---------+---------
    82975, +5172----11---------+72------4+---------+---------+---------+---------+---------+---------+---------
    82976, +---------+--72-----+---------+---------+---------+---------+---------+---------+--5------+---------
    82977, +---------+---------+---------+---------+---------+---------+--------1+---------+---------+---------
    82978, +---------+---------+XXX------+---------+---------+---------+6--------+---------+---------+---------
    82979, +5-6------+55-----255--5------+---------+---------+---------+5--------+---------+---------+---------
    82980, +---------+-4-7-----+---------+---------+---------+---------+---------+---------+---------+---------
    82981, +-----3616+---------+---------+---------+---------+---------+---------+---------+4--------+---------
    82982, +--66X54--+71-------+---------+---------+---------+---------+---------+--------443----1134+---------
    82983, +---------+--1------+-------33+---------+---------+---------+---------+--------52---------+-----2X--
    82984, +-4233----+---------+---------4X78------+-42------+---------+--1------+---------59----59--+---------
    82985, +---------+---61----+---------1XXX5998--+-52--2---+48928----+---------+---------+---------+--53-----
    82986, +---------+---------+---------+--------77---------+---37----+---------+---------+---------+---------
    82987, +-----6---+---------+---------4---82-5-X8-5-------+---------+---------+---------+---------+---------
    82988, +--17-3---+-----5---7---------+--------5XXX4------+------3--+-------6-53--------+---------+---------
    82989, +---315-4-+---------+---------+---------+-21X2---2X1--12----+----22---+---------+------24-+---------
    82990, +---------+---------+15-------11----7XXX67XX444---+---------+------45-+---------+----44--43---------
    82991, +---------+---------+---------+-------37XX3--73553X2X-------+---------+4X16-----+---------+1X-------
    82992, +4978-----+---------+---------+---------51--------+---45367-+---------+----5998-+716898988XX9989XX7-
    82993, +986197---+---5X7-4-+--8-5---6+---------+---------+---------+---------+------8X9+-1X6--6--+56-------
    82994, +---------+---------+--3------+---------+---------+---------+---------+---------+----3----+----74---
    82995, +---------+---------+---------+---------+---------+---------+---------+---------+---35----+---------
    82996, +---------+---------+---------+---------+---------+---------+---------+---------+---7XX3--2X1-63----
    82997, +---------+---------+---------+---------+--------1+--------42---------+---------+---51----+---------
    82998, +---------+---------+-----2---2---------+-------5-+---------+---------+---------+---7-----+---------
    82999, +-2-------+---------+---6483--+3--------85--------+--6------6-4-5-----+---------+-7--27---+9--------
    83000, +---------+---------+---------+-----3---+--15-----+---------+---------+---------+---------+---------
    83001, +---------+---------+------1522---------+------1--+---------+---------+---------+----3X844XXX9------
    83002, +---------+---X-----+--54-----+-------3-+---------+-------3-+---------+---------+6262----78---------
    83003, +---------+---------+---------+---------+---------+---------+---------+---------+----7XXX9+----5----
    83004, +---------+---------+---------+---------+----7----+---------+------32-+---------+7---6--286---------
    83005, +--1------+---------+X4-------+---------+---------+--------23---------1X3--5--1X3---------+---------
    83006, +2---523--+---------+-------25+32-------+---------+---------+---------+31----9--+-3-------+----2----
    83007, +---------+---------+---------+---------41--------+---------+-------11+---------+-3---81--+---------
    83008, +---------+---------+---------+---------+---------+---------+-----4---+---------+17-81X62-+8X5------
    83009, +-5-72----+---------+---------+---------+---------+---------+---------+---------+--35----41---23----
    83010, +---------+---------+---------+---------+---------+---------+--1--2414+-914-343-61-----4X31-99X---6-

[udoklein]

The really big issue is the really high jitter. This is really very nasty :(

[nameoftherose]

I think I will use eurochron clock as the fading detector - I will reset Leonardo if it fails to synchronize. Regarding idea 4c, could the bandwidth be defined at compile time? or even changed by user via the serial interface? I am pretty sure that you can safely ignore the «signal» during daytime. And of course you may leave the issue for somebody else to resolve.

[udoklein]

How do you want to do this?

With regard to defining a different bandwidth. Yes both is technically possible and not hard to do. The bandwidth is controlled here:

uint16_t N = ticks_to_drift_one_tick / bin_count; void set_has_tuned_clock() { // will be called once crystal is tuned to better than 1 ppm. N = tuned_ticks_to_drift_one_tick / bin_count; }

So if you remove the line in set_has_tuned_clock and implement an additional function you can vary the bandwidth anyway you want. The challenge is to determine when to vary the bandwidth and what would be the most suitable value. Also there is something else which is pretty nasty. If you lower the bandwidth the frequency control mechanism will receive way more jitter. This in turn may destabilize the frequency control. So you would better set "has_stable_ambient_temperature" to false.

I will start to experiment with these settings based on the logfiles you have given me.

[udoklein]

By now my analysis tool is in a somewhat reasonable form that I can publish it. You can find it here: https://github.com/udoklein/dcf77/releases/tag/v3.2.0 With this I hope to get better insights into your issue. In particular: if I can reproduce the observed behaviour with this tool I get all the benefits of using Linux. That is: unlike in the Arduino environment I can now very easily debug what's going on :)

[nameoftherose]

Thank you. What is 'Prediction Match Statistics'? Is quality factor between 0 and 1? I think I will feed the receiver from battery hoping to avoid power supply noise.

[udoklein]

The library does not decode the signal directly. Instead it synthesizes a signal locally and tries to maximize correlation with the received signal. PM = 50 would indicate perfect match. PM = 30 would indicate that the signal has already a lot of nice. PM = 0 would indicate that the signal (for whatever strange reason) has no noise but reverse polarity. The statistics are just the count over the whole received siganl.

Quality factor starts at 0 and increases as predictions match better. It is an approximately logarithmic measure for signal to noise ratio. So 3 would be already more than good under your conditions. Where I am located I can easily reach more than 5. I start to document the tool right now.

[udoklein]

Now I also added some documentation for the tool (a20c65158eb7b8c01163a009b9ca10de17e2b461) https://github.com/udoklein/dcf77/blob/a20c65158eb7b8c01163a009b9ca10de17e2b461/extras/standalone_debug_helper/README.md

@nameoftherose tell me what you think about it. Is this sufficient?

[nameoftherose]

A new log is added taken on an UNO on breadboard with crystal. You can find it in dcf_20170819T183813.zip If you are still interested in such measurements, please tell me and I will try to collect a longer continuous log.

[udoklein]

Yes, I am still interested although I have very limited time. This log is very interesting. In particular the part here:

Decoded time: 17-08-19 6 23:50:14 CEST ..
  17.08.19(6,6)23:51:13 MEZ 0,0 17 p(5825-6:255) s(195-0:27) m(239-0:33) h(247-4:34) wd(250-129:17) D(245-129:16) M(246-107:19) Y(250-133:16) 127,50,50,50
Clock state: synced
Tick: 16

    11575, +---------+---------+---------+---------+---------+---------+6XXXXXXXXXXXXXXXXXX6---------+---------

Decoded time: 17-08-19 6 23:50:15 CEST ..
  17.08.19(6,6)23:51:14 MESZ 0,0 17 p(5817-0:255) s(195-0:27) m(239-0:33) h(247-4:34) wd(250-129:17) D(245-129:16) M(246-107:19) Y(250-133:16) 127,50,50,50
Clock state: synced
Tick: 16

    11576, +---------+---------+---------+---------+---------+---------8XXXXXXXXX1---------+---------+---------
  ??.??.??(?,?)??:??:255 MEZ 0,0 0 p(0-0:0) s(6-4:0) m(0-0:0) h(0-0:0) wd(0-0:0) D(0-0:0) M(0-0:0) Y(0-0:0) 0,0,0,255
Clock state: useless
Tick: 7807
  ??.??.??(?,?)??:??:255 MEZ 0,0 0 p(0-0:0) s(6-4:0) m(0-0:0) h(0-0:0) wd(0-0:0) D(0-0:0) M(0-0:0) Y(0-0:0) 0,0,0,255
Clock state: useless
Tick: 8807

The phase lock is perfect and then out of blue sky it resets the lock. This is strange. I have no freaking idea why this happens.

[udoklein]

The behaviour in the emulator is different. This is anoying. As I said, I have no clue what is going on here.

[nameoftherose]

As I say in the logfiles/README.md

2017-08-19 23:50:15 last synced 01:55
then restart without synchronization due to power supply noise

At that time there was a restart (due to accidental loss of power). synced state was not reached afterwards because of power supply noise (ie I had to connect the tablet to its power adapter due to low battery) You may have noticed from the log that tick count and scope count reset at this time. I am sorry I was not clear. I have also noticed slight differences between the emulator and arduino, but I attribute them to the fact that the log does not start from the very beginning. You have being very cooperative, thank you!

[nameoftherose]

At long last, the promised continuous 48 hours log file (actually 58 hours) is posted here dcf_20170820T195715Z.zip In order to record it, I had to put the tablet on power outlet for a considerable amount of time (>50%), so the noise characteristics change. The clock performed significantly bettter when on battery. When on power outlet its performance was like that of the eurochron clock. I am sure that if I feed the arduino directly from a 4x1.2V battery pack its performance would be very good (but I could not collect the statistics necessary to prove it).

[nameoftherose]

With better filtering of the usb power supply lines, the clock (UNO on breadboard with crystal and Conrad module) can sync while connected to my desktop computer, so long logs can be recorded. When the clock starts a little earlier than the signal reappears (around 20:00) it will sync and provide accurate time for days. It syncs every night for 8-10 hours and during the day alternates between locked and unlocked and sometimes free state. These observations are shown in README and 178 hour log Crystal UNO Filtered USB power in sync for ~40% of time.
But if the clock starts a little after the start of a long period of no signal (eg at 09:00), it will not sync when the signal reappears. (Even worse it may sync to the wrong time, has a preference for August 2000).On the other hand if the clock is connected to battery it will sync. So my conclusion is that this issue, although real, is caused by noise coming over the usb power supply line. I have tried an LC filter but without further improvement.
The true conclusion of course is that a bigger antenna and/or better receiver is necessary.
On this evidence, this not a problem of the library, so this lssue may be closed.
Nevertheless, one could try to:

• reset the clock after a long period in useless/dirty state or year <17
• initialize the clock with an almost correct time
• implement your suggestion to change the threshold of what is considered a reasonable phase lock

To reset, one could of course reset the mcu, but I find that too heavy handed. Is there an function in the library that re-initializes the decoder? Preferably not resetting the counts and the local clock so that it remains in the free running state. Concerning the second idea, in the comments section of The Clock you mention that :

if a DS3231 would be used , then a reliable lock would be easily possible in 1-2 minutes even in the presence of noise

Is this functionality implemented? If not please suggest how I could add it (but please do not add it yourself).
Year == 00 is a reliable fading detector, could be used to reset the decoder, release the lock which in turn would purge the other filter stages immediately and be included in the calculation of the the Global Quality Indicator.

[udoklein]

I would still assume it is a library issue. Reseting it if it runs dirty for prolonged periods of time would be an option.

Unfortunately Year == 00 is not a reliable fading detector. It would be a reliabe detector if every module would behave like yours. However there are several types of modules. One type fades all 0. The other fades all 1. A reliable fading detector would require this information as a configuration flag. Or alternatively: if the phase unlocks either the signal is drowned in noise or fading.

The functionality for DS3231 is not implemented. There are two things to consider:

1) DS3231 could be used to create a "tuned" reference clock of 32768 Hz. Precise to better than 2 ppm out of the box. In particular the auto tune code could then be fully removed. This is not hard to implement but requires a little bit of thought.

2) DS3231 can be used to keep time independent of DCF77 with 2ppm accuracy. Or typically less than 1 minute in 300 days. Hence the convolutional filter could be fed from the DS3231 which would allow to reaquire a phase lock very fast.

3) Now the part that you are probably after: assuming that DCF77 was synced at least once and that the time was used to sync the DS3231 with the DCF77 time the following can be done:

1. If a phase lock is can be acquired withing 250 000 seconds since the last good signal AND if there was no possibility for a leap second in between, then the DS3231 phase can be resynced giving proper time.
2. If a phase lock AND a second lock can be achquired within 1 450 000 seconds (or 167 days), then the DS3231 phase and second can be resynced giving proper time. No matter if a leap second happened or not.
3. If a phase lock AND a second lock AND a minute lock can be acquired within 899 500 000 seconds or 28 years, then DS3231 phase, second and minutes can be resynced giving proper time. Again, no matter if a leap second happened or not.

Algorithm: DS3231 has a known drift and the DCF77 time must be less than the maximum possible drift away from DS3231 time. Hence Set the DS3231 time (phase, seconds or minutes) according to what is received from DCF77. Then set the next digit (seconds, minutes or hours) +/- 1. Then you have three possible candidate times. Now pick this of the 3 which has the least difference to the current DS3231 time.

If the DCF77 signal is clean, then another indicator is: if the received DCF77 signal matches the predicted time (derived from DS3231) for one minute, then (modulo phase) a resync is achieved.

The point is that such a resync does not require a full DCF77 sync anymore.

[nameoftherose]

Thank you. I must study your answer and the library sources very carefully.

[udoklein]

Yes and no. The behaviour is more or less identical however the log files of the debug helper usually starts its output only after some time. Thus feeding the simulator from the debug log will always give different results then the real clock. This is not an issue of the simulator but of the log mechanism. I know this is not optimal but so far I did not find the time to modify the debug helper accordingly.

Since the library passes the travis tests https://travis-ci.org/udoklein/dcf77 I would say either your vi is broken or you deleted it accidentally.

The compiler as well as my editor agree that the braces are matching.

[udoklein]

I neither use vi nor emacs except if I have no other choice. If some editor behaves weird I do not see why I should know anything about it.

[nameoftherose]

Referring to your comment of Sep 9 and as far as DS3231 is concerned, I was merely looking for a method to manually set the time and clock state. It seems this can be achieved using DCF77_Local_Clock::setup() to set the state and defining a new DCF77_Encoder::setup(...) method to set local_clock_time to the desired time. (unfortunately it is not of much help with fading) If I were to add a TXCO to the clock, I would rather use it for the ATMEGA328 clock, or for timer2 external clock. Thanks for pointing out DS3231 high accuracy, its OSC output can be used as a very stable oscillator. Pity it cannot provide the 1ms interrupt for the library.

[udoklein]

The use of a DS3231 as a clock for the ATMEGA is what I had in mind. Obviously it can generate a 32768 Hz signal of high stability. 1 ms = 32 + 768 / 1000 cycles. Thus Bresenhams algorithm could be used in a suitable ISR to bring this to the desired 1 ms cycle length. In order to get rid of complications this should only be used WITHOUT the autotune functionality. Since the DS3231 is very stable anyway tuning it would be somewhat pointless anyway. That is the filter can narrow the bandwidth without tuning immediately from the start.

Using it to keep time for fast startup is in fact another option which I did not have in mind. Nevertheless this is a very reasonable idea.

[udoklein]

As am afterthought I would say that a DS3231 based clock design should be completely different. If it is acceptable to set the time at least once then it would be sufficient to just phase lock DS3231 (to the minute not to the seconds) and keep track of the leap seconds. Anything else (timekeeping, short term frequency stability) would be provided by DS3231 out of the box.

The point that I am trying to make: I already considered adding a DS3231 but I always find it strange to add a dedicated time keeping chip. My decoder was designed for a crystal. As soon as I start adding more features / hardware support I always come to the conclusion that I should better switch to a cheap wireless chip (e.g. DS2866) and grab time from a dedicated ntp server.

[nameoftherose]

I broadly agree with you concerning DS3231. It appears I have made some progress regarding the fading problem. The library is shining, Eurochron has not synchronized for more than half of the corresponding days. (But my code is ugly) (A 26 character timestamp is prepended to each line of the logfile. sed -r 's/^.{26}//' will remove it). (have been unable to locate DS2866, you do not mean ESP8266?)

[udoklein]

Cool, that's the way open source is supposed to work. I did not find the issue now you are fixing it on your own. Once your are satisfied with your changes I will review them and merge them back into the main line :)

If you want to I could also provide some experimental branch for the DS3231 as a crystal substitute.

[nameoftherose]

The Clock is operating continuously for 31 days. It has synced every night without ever decoding a wrong time. I intend to keep the clock running with the current code until a power cut occurs. I will not collect logs any more. But the year <= 0x17 criterion is a hack, there is no logic supporting it and probably works with the Conrad module only. Do you think it could be replaced by a date_parity test?

[udoklein]

Unfortunately date has even parity. Thus if the signal goes flat to 0 then the date's parity will be indicate proper parity although the date is definitely wrong :(

[nameoftherose]

I have never observed a 00-00-00 date. I think that you already ensure that month, day and weekday are valid. My thought was to use date_parity as a consistency check for all date fields combined. Is the parity calculation expensive? Is the date_parity to be relied upon? Does the library already use the parity bits? In what way? Will you please provide a log file for bad period in your location? Does it show jitter?

[udoklein]

You might want to read this: https://blog.blinkenlight.net/experiments/dcf77/the-clock/. Starting from there you can read the history and the ideas behind my decoder. The basic idea is to NOT DECODE at all. Instead correlate the received signal with potential signals and keep the one that matches best. This implicitly evaluates the parity bits. Hence it is somewhat pointless to evaluate the parity unless I would have significantly more memory available (compared to the whimsical 2k of the Arduino). Hence I do not evaluate the date parity. Keep in mind that the signal is suppsed to be noisy.

[nameoftherose]

I have already read it and more than once. I suspected that you do not use date_parity, (this is not a criticism, I do understand that parity check offers little given the robustness of your approach), I wanted to confirm. But why you say 'it is somewhat pointless to evaluate the parity'? The Flag_Decoder does so and the get_date_parity() function seems to return its value (but it is never called).
The signal is noisy, isn't just supposed to be. My chief concern is that if I try to use date_parity I may reject a lot of valid dates.

[udoklein]

Parities are implictly evaluated due to the statistical nature of the decoder. However this discussions led me to an interesting idea. I could add additional 1-2 additional decoder bins for decoding flat signals. This in turn would allow detect flat signals and thus reject them.

I think this would also resolve your issue. Might take me a little bit to implement though.

The reason why I never considered this is the additional memory + runtime consumption. With the "streaming" approach that I introduced some time ago only the memory issue remains. But I think this should be tolerable.

What do you think?

[nameoftherose]

Is the streaming approach documented somewhere? If that way one would be able to distinguish between no signal an signal buried in noise that would be an interesting proposition. But it would not make much difference in practice, the year errors appear with p.qf=255, s.qf>20, that's plenty of signal. Are the quality factors broadly equal in your location? Here the date quality factors are much lower (~25 vs >30) and it takes much longer to reach the final values. There are places in the house where the Clock stays synced (running on batteries) for more than 20h a day.

[nameoftherose]

Here is the latest version for your review. dcf77_2017-12-16_1753.src.zip It works satisfactorily in my place with the Conrad module.

[udoklein]

Hopefully the latest patch (see the long discussion here: #31) has resolved all this issues. I close this issue for now.

[nameoftherose]

As a matter of principle, I report that the clock, running my latest modifications, synced yesterday to the wrong time and stayed there for about 30 minutes. The error was in the minutes field and and was nearly 12min.

[udoklein]

Sorry for my slow response. I was 6 weeks offline due to issues with my internet provider. Today I finally got access again.

Do you have a trace for me? If not, did it sync from a proper time to the wrong time or did it sync to the wrong time after a cold start?

[nameoftherose]

There is no reason to be sorry. It synced to the wrong time while starting after a cold start. I have a log, but it does not start from the cold start and does not contain the signal ( I created it in order to record the transition to summer time). And it was produced by a modified version of the library and Swiss_Army_Debug_Helper that is based on the leonardo support version of yours. So I thought it would not make sense for you to spend any time on it. Anyway, since you asked, here it is: dcf20190330.zip Thank you.

[nameoftherose]

I had a nasty surpise today, decoded time totally crazy. If you have any thoughts from just looking at it, please advice. Do not waste more than one minute. The datetime was still crazy after it synced again although the decoder stages predict datetime correctly. I hope it is a memory glitch. Thank you.

[2020-03-17 16:09:16:043] locked: 20415-1514-01 18:115:00 CEST (UTC+2) [2020-03-17 16:09:16:043] set mode: w [2020-03-17 16:09:17:069] useless: 5889 dirty: 357 free: 9916 unlocked: 159918 locked: 225688 synced: 892464 Time of 1st Sync: 20-03-02 1 18:22:51 CET .. Time of last transition to Sync: 20-03-15 7 19:05:43 CET .. Time of last transition from Sync: 20-03-17 2 10:21:17 CET .. [2020-03-17 16:09:17:070] set mode: a [2020-03-17 16:09:18:057] confirmed_precision ?? adjustment, deviation, elapsed [2020-03-17 16:09:18:059] 0.5000 ppm, @- , -23.1250 ppm, -3 ticks, 553 min + 19298 ticks mod 60000 [2020-03-17 16:09:18:059] 41/100~230/600 [2020-03-17 16:09:18:059] set mode: d [2020-03-17 16:09:19:031] [2020-03-17 16:09:19:073] Decoded time: 4F-FE-01 2 18:1F:00 CEST *L Clock StateFlags: DateParity=-127 UsesSummerTime=-127 LeapSecond=-5 TZChange=-8 Year Error Count: 231 Week Day Inconsistency Count:[2020-03-17 17:09:19:075] 61 20.03.17(2,2)16:10:18 MEZ 0,0 0 p(61583-2940:255) s(194-35:22) m(175-109:9) h(220-150:9) wd(78-66:1) D(230-196:4) M(122-98:3) Y(1[2020-03-17 17:09:19:076] 22-116:0) 127,8,5,38 [2020-03-17 16:09:19:076] Clock state: locked [2020-03-17 16:09:19:076] Tick: 27

[udoklein]

This should never happen. I would agree that this is some kind of memory glitch. I had something similar once when I was running out if memory and the stack smashed the static allocated memory.

[nameoftherose]

Hello after so long time. May I ask what prompted you to implement the «more conservative demodulator handling» modification?

[udoklein]

One user of the library reported a very strange behaviour. And he provided logfiles that allowed me to nail down what was going on. The issue is somewhat complicated, I try to explain it.

First of all we have to acknowledge that this library is pushing the limits of what can be done with 2k of main memory and very little processing power. Hence the underlying noise modell is "AWG" or "average white gaussian" noise. Which is of course not correct from a theoretical point of view.

As it turned out it was not that bad though. And the library was fine in the first place. But then some users complained about slow lock times and I lowered the noise threshholds. This had the effect that under very specific circumstances the library would lock to the noise and decode a signal which was completely out of phase. Due to the "auto tune" feature the library then found that the crystal was completely off the mark and tried to compensate this. Then the library would drift out of phase (but still believe to be in phase) and the noise would rise (due to the decoder being out of phase) and the library would lose the lock. At that point he crystal was completely wrong tuned. So no further lock would happen and the library would not reaquire any further lock. As a consequence it would continue with the flawed time on a mistuned crystal.

I never encountered the issue while testing and it will not affect most users of the library. But I remember that some others also complained about strange behaviour.

Now the challenge is how to fix it? As I said I have only 2k. If there would be more memory (say 16k or 64k) and more CPU (e.g. ARM based controller), then this would be very easy to fix. The easiest fix would be to run several decoder instance in parallel with different tuning parameters and also with offset local clocks on purpose (think "genetic algorihtm"). However I neither have 10 times memory nor 10 times CPU. Hence I just increased the thresholds. This slows down initial lock time. This is an issue for you if you have very bad reception with lots of fading. However if you have good reception but lots of background noise (e.g. closer to the AWG model) then this does not really matter.

So I had to make a choice: allow it to lock to the wrong signal for some users or accept that some will not get locks anymore. I decided that wrong decoding is worse than not decoding at all. Hence the increased thresholds.

Is this somewhat understandable?

[nameoftherose]

Yes it is thank you. I am in 'lots of fading' location, all of this applies to me.

[nameoftherose]

You say "At that point the crystal was completely wrong tuned. So no further lock would happen and the library would not reaquire any further lock. As a consequence it would continue with the flawed time on a mistuned crystal.". For this, I have additionally modified lines 1572,1573 in dcf77.cpp to read:

       if (total_adjust >  max_total_adjust) { total_adjust = 0/* max_total_adjust*/; }
       if (total_adjust < -max_total_adjust) { total_adjust = 0/*-max_total_adjust*/; }

Seems to work, but it is very difficult to be sure as this happens very rarely. Do you see any trap with this? Are there any other DCF77_Frequency_Control state variables that also have to be reset?

[brettoliver]

Hi Udo. Thanks for all your work on this issue. This sounds like the issue I was having some time back. Not sure what you mean by signal fading but I have serveral clocks running your decoder all fed off the same aerial via digital repeaters. I noticed after a long priod of very noisy signal some clocks would tune to the wrong time and other would not(they all had bit for bit the same signal).

At the time some clocks were running different versions of your library and never tuned to the wrong signal. I think at the time you made some changes to the library as a workaround.

When I get some spare time I will try and test out your new version side by side with the old.

Not sure if it's useful for anyone based in the South of the UK but I send live basic signal reception info 24/7 so I can monitor my clocks if they error to see if signal noise was a problem. The charts show 1 = 0 signal match errors over a minute or 0 = any signal match errors at all in the last minute

Last hour https://thingspeak.com/channels/196020/charts/7?bgcolor=%23ffffff&amp;color=%23d62020&amp;dynamic=true&amp;results=60&amp;title=DCF77+Signal+Match+Last+Hour&amp;type=line&amp;yaxismax=1&amp;yaxismin=0

Last 6 hours https://thingspeak.com/channels/196020/charts/7?bgcolor=%23ffffff&amp;color=%23d62020&amp;dynamic=true&amp;results=360&amp;title=DCF77+Signal+Match+6+Hour&amp;type=line&amp;yaxismax=1&amp;yaxismin=0

Last 2 days https://thingspeak.com/channels/196020/charts/7?bgcolor=%23ffffff&amp;color=%23d62020&amp;dynamic=true&amp;results=4000&amp;title=DCF77+Signal+Match+2+Days&amp;type=line&amp;yaxismax=1&amp;yaxismin=0

May be useful to see if your signal is erroring and mine is not it may be local interrerance.

I use this site as a ref for my signal reception https://www.dcf77logs.de/live

[udoklein]

The code

       if (total_adjust >  max_total_adjust) { total_adjust = 0/* max_total_adjust*/; }
       if (total_adjust < -max_total_adjust) { total_adjust = 0/*-max_total_adjust*/; }

has two pitfalls.

1) It will not really work if the crystal has a deviation of let's say 126 ppm. However if your crystal is reasonable good you will never encounter this situation.

2) If you reset the adjustment you also need to reset everything that is tripped when it gets tuned. That is you have to undo whatever happens in

        static void on_tuned_clock() {
            if (Configuration::has_stable_ambient_temperature) {
                // If ambient temperature is not stable tuning
                // the crystal is no guarantee for reasonable
                // accurate local frequency. Hence we will
                // propagate the event only if ambient
                // temperature is considered stable.
                Demodulator.set_has_tuned_clock();
                Local_Clock.set_has_tuned_clock();
            }
        };

In particular you should call all of the reset_has_tuned_clock() functions.

[nameoftherose]

Thank you. I am grateful. I am studying what you say and I will implement them at the soonest. Concerning point 1, yes I have 10ppm crystals, the usual adjustment range is between 30 - 35 ppm and precision better than 0.5ppm.

[nameoftherose]

In the Swiss_Army_Debug_Helper's Scope namespace you have a count variable that gives the total seconds since boot. The difference Decoded_Time-count points to boot time and therefore is invariant. In principle it could be used as an overall datetime correctness / consistency criterion. But I see several issues: 1. The difference might not be calculable within the time constrains of the isr (especially with atmega 328) 2.The invariance breaks down in case of leap seconds and DST change over. 3. It seems that when Clock_Controller::Demodulator.Binning::Convoluter<uint16_t, Clock_Controller::Configuration::phase_lock_resolution>::signal_max_index wraps around 0 or 99 a second is missed or added. I would appreciate any opinions on this idea.

[udoklein]

Well, this could be used. However the issue is somewhat deeper. Let's look into why this decoder exists and was implemented that way. My idea was to achieve the following:

1) Learn about statistical decoding. 2) Push the limits on what can be done with a somewhat whimsical controller (e.g. Arduino). 3) Provide a superior DCF77 decoder.

I think (1) and (2) were achieved 100%. However with regards to (3) I have this issue that the decoder behaves somewhat bad under very noisy conditions. Typically under conditions that make other decoders just fail.

The question is how to fix the decoder behaviour? One approach is adding something like you suggest. However I am quite sure that there will be edge conditions were this will also fail.

The other approach would be to fix this completely by implementing a better decoder from what I learned with goal #1. The most important insight was that my noise model (AGW) is wrong. Actually DCF77 reception has some AGW part but also (in particular under bad reception conditions) there are just long and continued outages were it picks up only noise. To make matters worse the noise is biased depending on the demodulator.

So if I would be going to fix it, then I would implement the following changes.

A) Custom demodulator. B) Also decoding the "hidden" FM signal C) Use a more powerful controller (e.g. STM32), in particular more cycles, more memory

However this begs the question if there is any usecase that would justify this effort. Let's see.

Goal #1 --> nope, I learned the lesson, I doubt that I will gain any additional insights. Goal #2 --> nope, I actually learned that the AVR family is a little bit to whimsical for more sophisticated algorithms Goal #3 --> this would be the only remaining goal.

But why should I push #3? Well, an accurate and noise resilient wireless clock. N.B. low power consumption is not on the list as my DCF77 decoder is not really low on power consumption. Now wireless and noise resilient are quite easy to achieve. Just use a ESP8266 and sync by NTP. Or use a cheap GPS module. Or setup a RASPI with time keeping GPS and distribute the time by whatever protocol to the clock.

Having said that: I agree that this approach might provide some improvement but it I am not going to implement it. The effort vs. gain on my side is to high to justify pushing this any further.