search

JvanKatwijk / eti-stuff

experimental software for creating and interpreting eti frames
GNU General Public License v2.0
21 stars 12 forks source link

ETI output in SYNCH #26

Closed lars18th closed 6 years ago

lars18th commented 6 years ago

Hi @JvanKatwijk ,

Please, read my request at https://github.com/Opendigitalradio/dabtools/issues/16

I suggest to add a proper "padding" output when the no-signal or insufficient data for decoding exists. This will provide the option for generate a CBR ETI bitstream that will be more easier to process in the "eti-backend" (aka DAB player).

You agree?

JvanKatwijk commented 6 years ago

Reading the discussion it seems there is no need to pad, since either eti frames are transmitted completely or not at all. The backend then - obviously - has to keep track of the frames so that it can detect missing frames, but since eti frames are largely selfcontained that is not a big issue

2018-02-16 12:54 GMT+01:00 lars18th notifications@github.com:

Hi @JvanKatwijk https://github.com/jvankatwijk ,

Please, read my request at Opendigitalradio/dabtools#16 https://github.com/Opendigitalradio/dabtools/issues/16

I suggest to add a proper "padding" output when the no-signal or insufficient data for decoding exists. This will provide the option for generate a CBR ETI bitstream that will be more easier to process in the "eti-backend" (aka DAB player).

You agree?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JvanKatwijk/eti-stuff/issues/26, or mute the thread https://github.com/notifications/unsubscribe-auth/AITzwCyiQ8sM_mHJPrwqWVplI7lW5QEbks5tVWxagaJpZM4SIOoZ .

-- Jan van Katwijk

+31 (0)15 3698980 +31 (0) 628260355

lars18th commented 6 years ago

Hi @JvanKatwijk ,

Please read my disection of the ETI-NI standard regarding this at https://github.com/Opendigitalradio/dabtools/issues/16#issuecomment-366256004

I think the documentation explicitly explains what needs to be done to send NULL transmissions. See section 5.9 of Page 27: http://www.etsi.org/deliver/etsi_i_ets/300700_300799/300799/01_30_9733/ets_300799e01v.pdf

In fact, this it's required to deliver correct ETI-NI frames. And it's quite easy to implement as I exposed (more or less, catch the exception of "no signal" [aka, no lock, impossible to decode, etc.] and generate a NULL packet).

I hope you'll like to implement it! :smile:

JvanKatwijk commented 6 years ago

I have read it. It implies that if there is no signal, null frames are to be sent out. Impossible with the current structure: handling data is hard coupled to receiving synchronized data, so if there is no sync, the processing chain is just waitting for data to arrive

You generate a lot of messages, what is the objective, since I do not get the feeling that you want to contribute yourself

2018-02-16 15:59 GMT+01:00 lars18th notifications@github.com:

Hi @JvanKatwijk https://github.com/jvankatwijk ,

Please read my disection of the ETI-NI standard regarding this at Opendigitalradio/dabtools#16 (comment) https://github.com/Opendigitalradio/dabtools/issues/16#issuecomment-366256004

I think the documentation explicitly explains what needs to be done to send NULL transmissions. See section 5.9 of Page 27: http://www.etsi.org/deliver/ etsi_i_ets/300700_300799/300799/01_30_9733/ets_300799e01v.pdf

In fact, this it's required to deliver correct ETI-NI frames. And it's quite easy to implement as I exposed (more or less, catch the exception of "no signal" [aka, no lock, impossible to decode, etc.] and generate a NULL packet).

I hope you'll like to implement it! 😄

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JvanKatwijk/eti-stuff/issues/26#issuecomment-366258478, or mute the thread https://github.com/notifications/unsubscribe-auth/AITzwM8OEbCw3eOrFQC7kr89c-SuW6clks5tVZfVgaJpZM4SIOoZ .

-- Jan van Katwijk

+31 (0)15 3698980 +31 (0) 628260355

basicmaster commented 6 years ago

@lars18th: I indeed forgot about § 5.9 and the null transmission, and therefore added support for it to DABlin, to prevent any issues in case it occurs.

However I still don't see the actual reason why generating such ETI frames should be generated here or by the dabtools.

JvanKatwijk commented 6 years ago

It would require a redesign since generating eti data only takes place when/if we can interpret the signal, i.e. have a decent sync. The front end drives the rest, so if the front end is not in sync the rest is waiting

2018-02-16 18:52 GMT+01:00 Stefan Pöschel notifications@github.com:

@lars18th https://github.com/lars18th: I indeed forgot about § 5.9 and the null transmission, and therefore added support for it to DABlin, to prevent any issues in case it occurs.

However I still don't see the actual reason why generating such ETI frames should be generated here or by the dabtools.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JvanKatwijk/eti-stuff/issues/26#issuecomment-366309227, or mute the thread https://github.com/notifications/unsubscribe-auth/AITzwJV6utn-96kuk-9HnkGFzjc4GI95ks5tVcBZgaJpZM4SIOoZ .

-- Jan van Katwijk

+31 (0)15 3698980 +31 (0) 628260355

lars18th commented 6 years ago

Hi @JvanKatwijk ,

You generate a lot of messages, what is the objective?

The objective it's have the tools to decouple the DAB demodulator from the DAB decoder. So I need to have ETI source bitstreams (from satellite, from ODR-DabMod, or from "eti-cmdline-*"); and ETI consumers (like DABlin).

The main problem at time it's to standardise (in the open source community for DAB) the ETI as an interchange format. For example, at time onle two ETI players exists: DABlin and "eti-backend".

Hi @basicmaster ,

However I still don't see the actual reason why generating such ETI frames should be generated

The reason it's to improve the compatibility with any DAB decoder. If some LI frames are missed from the ETI bitstream, then the decode process will be more complex.

Hi @JvanKatwijk ,

It would require a redesign since generating eti data only takes place when/if we can interpret the signal

OK. Now I know about the problem. So, I see two "simple" solutions that can be applyed:

  1. If a discontinuity is created then increment the COUNTER of the next delivered frame. This will permit to the decoder to see the discontinuity. And in order to detect the discontinuity you can use a simple var with last timestamp (aka previous frame), and compare with the current timestamp (aka current frame). Then compare if more than 24ms (*N) time is ellapsed.

  2. The second option it's wait to interpret the signal with a timeout. If more than 24ms are passed, then generate an empty frame and continue waiting for reading with the same timeout.

I know that this change isn't trivial. But it enables the use of the ETI format as an interchange format. What you think?

lars18th commented 6 years ago

Hi @JvanKatwijk ,

As I described here http://github.com/AlbrechtL/welle.io/issues/217, at time, I'm using a brute-force-workaround to do the decoupling of the DAB demodulator and the DAB decoder. The trick is use the ODR-DabMod tool for generating the IQ samples and a custom RTL_TCP server that reads these IQ samples and sends to the DAB player.

However, I have a problem: The implementation of the ODR-DabMod requires no missing ETI-NI frames. So, I found that extending the eti-cmdline with a safe output (empty or NULL frames if no signal or huge errors) can be a solution.

What you think about this?

JvanKatwijk commented 6 years ago

How would you recognize that frames are missing

2018-02-26 10:17 GMT+01:00 lars18th notifications@github.com:

Hi @JvanKatwijk https://github.com/jvankatwijk ,

As I described here http://github.com/AlbrechtL/welle.io/issues/217 https://github.com/AlbrechtL/welle.io/issues/217, at time, I'm using a brute-force-workaround to do the decoupling of the DAB demodulator and the DAB decoder. The trick is use the ODR-DabMod tool for generating the IQ samples and a custom RTL_TCP server that reads these IQ samples and sends to the DAB player.

However, I have a problem: The implementation of the ODR-DabMod requires no missing ETI-NI frames. So, I found that extending the eti-cmdline with a safe output (empty or NULL frames if no signal or huge errors) can be a solution.

What you think about this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JvanKatwijk/eti-stuff/issues/26#issuecomment-368436434, or mute the thread https://github.com/notifications/unsubscribe-auth/AITzwLQbet1yQJU-cZf6o-BK0b_Ib6pZks5tYnakgaJpZM4SIOoZ .

-- Jan van Katwijk

+31 (0)15 3698980 +31 (0) 628260355

lars18th commented 6 years ago

How would you recognize that frames are missing

These are the three cases that I detected during my tests related to troubles with missing frames:

Please, note that the only reason to propose to generate NULL frames isto solve these particular cases!

Then how would we recognize that there are missing frames?

I'm not a DAB expert. But here is my proposal:

I don't know if this it's hard to implement or not. But I like to comment with you about if this helps or not to solve the problems.

What you think?

JvanKatwijk commented 6 years ago

That is not what I mean.

apart from errors in the eti bitstream, is there a clock in the accepting software or is it a counter for the eti frames

2018-02-27 10:24 GMT+01:00 lars18th notifications@github.com:

How would you recognize that frames are missing

These are the three cases that I detected during my tests related to troubles with missing frames:

  • Discontinuity detection by the ETI parser.
  • Error in the ETI bitstream.
  • Silence added to the output of the ETI parser.

Please, note that the only reason to propose to generate NULL frames isto solve these particular cases!

Then how would we recognize that there are missing frames?

I'm not a DAB expert. But here is my proposal:

  • Case 1: Errors in the Input. In case of the demodulator detects uncorrectable errors, then it needs to try to resynch until a good signal is detected. In the meantime, the output of the demodulator should be NULL frames.
  • Case 2: Discontinuity in the Input. When the demodulator detects that a hole exists in the input, it should "repair" the hole sending a NULL frame to the output in between the other correct frames.
  • Case 3: Input missing. When no input is detected, the output will be NULL until a signal is found.

I don't know if this it's hard to implement or not. But I like to comment with you about if this helps or not to solve the problems.

What you think?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JvanKatwijk/eti-stuff/issues/26#issuecomment-368804782, or mute the thread https://github.com/notifications/unsubscribe-auth/AITzwCmmbBjESvmFBIOkfpKNhpdgz21Gks5tY8m9gaJpZM4SIOoZ .

-- Jan van Katwijk

+31 (0)15 3698980 +31 (0) 628260355

lars18th commented 6 years ago

apart from errors in the eti bitstream, is there a clock in the accepting software or is it a counter for the eti frames

This varies according to the implementation. However, as far as I can deduce, the DAB Player mainly uses a clock for the audio output, and a counter for the incoming bitstream. So, as the ETI-NI has a counter (the FSYNC 24bit counter generated by the sender, that is included in the SYNC header, the 4 initial bytes of each ETI-NI frame) the only requirement is to generate the NULL frames with the correct counter mark. If this is done correctly the DAB decoder (aka Player) can have a deterministic behaviour even if the ETI frames have errors.

JvanKatwijk commented 6 years ago

are you sure that the only thing then that is to be included in inserted frames is the "correct" frame counter? and all other fields set to zero?

2018-02-27 11:29 GMT+01:00 lars18th notifications@github.com:

apart from errors in the eti bitstream, is there a clock in the accepting software or is it a counter for the eti frames

This varies according to the implementation. However, as far as I can deduce, the DAB Player mainly uses a clock for the audio output, and a counter for the incoming bitstream. So, as the ETI-NI has a counter (the FSYNC 24bit counter generated by the sender, that is included in the SYNC header, the 4 initial bytes of each ETI-NI frame) the only requirement is to generate the NULL frames with the correct counter mark. If this is done correctly the DAB decoder (aka Player) can have a deterministic behaviour even if the ETI frames have errors.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JvanKatwijk/eti-stuff/issues/26#issuecomment-368823666, or mute the thread https://github.com/notifications/unsubscribe-auth/AITzwCY78xEdhlf-xUnZ3VC3fc7-usEhks5tY9kcgaJpZM4SIOoZ .

-- Jan van Katwijk

+31 (0)15 3698980 +31 (0) 628260355

lars18th commented 6 years ago

are you sure that the only thing then that is to be included in inserted frames is the "correct" frame counter? and all other fields set to zero?

I explained here: https://github.com/Opendigitalradio/dabtools/issues/16#issuecomment-366256004

As a summary for NULL frames:

JvanKatwijk commented 6 years ago

I am not asking whether or not you explained it elsewhere. You are more or less commanding me to do something - which I might or might not - so it seems not more than polite to present me the spec for that

So again, can you specify what the structure/content of the frame to be generated is (and I am not going to look into what you might or might not have explained elsewhere, just a spec)

2018-02-27 11:51 GMT+01:00 lars18th notifications@github.com:

are you sure that the only thing then that is to be included in inserted frames is the "correct" frame counter? and all other fields set to zero?

I explained here: Opendigitalradio/dabtools#16 (comment) https://github.com/Opendigitalradio/dabtools/issues/16#issuecomment-366256004

As a summary for NULL frames:

  • 0x 0nnn FFFF FFFF FFFF FFFF ... 5555 5555 5555 (In HEX separated each 4 digits with spaces) -- The first '0' byte is the STAT field indicating a "full error"; -- Where nnn is the SEQ number generated by the sender; -- At leat 16 bytes with '0xF' values in the LIDATA; -- And rest until completing the size of the ETI-NI packet setting to '0x55'.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JvanKatwijk/eti-stuff/issues/26#issuecomment-368829852, or mute the thread https://github.com/notifications/unsubscribe-auth/AITzwJTa6If9sgk0tCF7jzMr0tO4xhZaks5tY95DgaJpZM4SIOoZ .

-- Jan van Katwijk

+31 (0)15 3698980 +31 (0) 628260355

lars18th commented 6 years ago

Hi @JvanKatwijk ,

I'm sorry! I want to be a gentleman. I've probably misinterpreted what you wanted.

In order to understand your request: You like to know the exact structure and content of a NULL frame? Is this or something else?

lars18th commented 6 years ago

Hi @JvanKatwijk ,

I see you got the idea and started an experimental version: https://github.com/JvanKatwijk/eti-stuff/commit/d8d38264be280bd6c2d9fce61b92c1ab29551863

Because I would like to collaborate (as much as I can)... please can you explain something about the changes? In the code I can see: "It will try to insert missing frames as null frames, starting with an error code 0 (i.e. non repairable)." So, this version will generate one NULL frame each time it detects "non repairable" data? This change grants a CBR in the output? That is 6144bytes each 24ms?

My idea is do some tests and report the results. You agree?

lars18th commented 6 years ago

Hi,

Regarding the SYNC output I have this new idea based on only two changes:

So, the concept is: the ETI generator works all the time in a push mode. The constant bitrate in the output is granted when the input driver is running, even if the incoming data is lost or erroneous.

You feel this is a feasible solution?

lars18th commented 6 years ago

Hi,

For the first goal (NULL frames from the ETI generator) my testing environment will be this:

Then when this file be processed by the eti-cmdline-sdr the expected output would be: Regular ETI frames at start, and when the "silence" appears a sequence of NULL ETI frames. And after the silence finishes the ensemble continues without any interruption. So, the output ETI-NI has a CBR and it's playable without errors in the backend (or DABlin).

I'm right?

JvanKatwijk commented 6 years ago

It means writing the ETI generator as a separate clock driven thread, which brings interesting synchronization issues with the ofdm handler. It is a complete rewrite/redesign, no one is stopping you to do it, but do not expect me to do it.

2018-03-07 13:08 GMT+01:00 lars18th notifications@github.com:

Hi,

Regarding the SYNC output I have this new idea based on only two changes:

-

First, the ETI generator will be modified to generate NULL frames when the input is NULL. The input is the data comming from the input driver. If this data is zero, null, have uncorrectable errors or are out of synch, then is detected as NULL and correspondingly one NULL ETI frame is output.

The second change is to add and internal clock to the input driver. That's more simple than sounds: the thread that reads samples from the driver saves the timestamps of last reads. If any drift passes a limit then it flush the input driver with zeroes. This will ensure that the input driver will output data in synch.

So, the concept is: the ETI generator works all the time in a push mode. The constant bitrate in the output is granted when the input driver is running, even if the incoming data is lost or erroneous.

You feel this is a feasible solution?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/JvanKatwijk/eti-stuff/issues/26#issuecomment-371118758, or mute the thread https://github.com/notifications/unsubscribe-auth/AITzwJgEi6ODeWj-W6X5RdVk4RiuXeYyks5tb83bgaJpZM4SIOoZ .

-- Jan van Katwijk

+31 (0)15 3698980 +31 (0) 628260355

lars18th commented 6 years ago

Hi @JvanKatwijk ,

It means writing the ETI generator as a separate clock driven thread...

I feel I'm not explained well for the first goal: The ETI generator will continue to be the same as now, working in push mode. The only difference is when it receives bad data... and not when it not receives data. I'm not an expert of the current code (as you can imagine), and it's possible I miss something. However, if the lower layers that send data to the ETI generator will pass zero data (all zeroes) when bad data is detected, then is possible to generate the NULL ETI frame.

Have you considered the example I mentioned?

JvanKatwijk commented 6 years ago

That is impossible, since the "low layers" are not aware of what is going on further. They have to be rewritten completely. Just look at the code

2018-03-08 9:18 GMT+01:00 lars18th notifications@github.com:

Hi @JvanKatwijk https://github.com/jvankatwijk ,

It means writing the ETI generator as a separate clock driven thread...

I feel I'm not explained well for the first goal: The ETI generator will continue to be the same as now, working in push mode. The only difference is when it receives bad data... and not when it not receives data. I'm not an expert of the current code (as you can imagine), and it's possible I miss something. However, if the lower layers that send data to the ETI generator will pass zero data (all zeroes) when bad data is detected, then is possible to generate the NULL ETI frame.

Have you considered the example I mentioned?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JvanKatwijk/eti-stuff/issues/26#issuecomment-371413617, or mute the thread https://github.com/notifications/unsubscribe-auth/AITzwIdsadD9Segc_yOKJFEsI3-YDzlAks5tcOlsgaJpZM4SIOoZ .

-- Jan van Katwijk

+31 (0)15 3698980 +31 (0) 628260355

lars18th commented 6 years ago

Hi @JvanKatwijk ,

Thank you for continuing to help me! :smile:

Just look at the code

I'm doing it, but I feel it difficult. Some issues for clarifying to me:

It's this correct?

JvanKatwijk commented 6 years ago

If you think it is difficult then do not start with a major rewrite

2018-03-08 9:57 GMT+01:00 lars18th notifications@github.com:

Hi @JvanKatwijk https://github.com/jvankatwijk ,

Thank you for continuing to help me! 😄

Just look at the code

I'm doing it, but I feel it difficult. Some issues for clarifying to me:

It's this correct?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JvanKatwijk/eti-stuff/issues/26#issuecomment-371422722, or mute the thread https://github.com/notifications/unsubscribe-auth/AITzwNLm_XHfwYcnd4oxz9IASKazpdroks5tcPJpgaJpZM4SIOoZ .

-- Jan van Katwijk

+31 (0)15 3698980 +31 (0) 628260355

lars18th commented 6 years ago

Hi @JvanKatwijk ,

If you think it is difficult then do not start with a major rewrite

I feel I can handle it! :smile: Regarding the first goal (generation of ETI NULL frames when the incoming data is invalid):

So assuming that the OFDM processor doesn’t miss packets (that isn’t true now, for exemple when the SYNC packet 0 isn’t detected or SYNC is lost)...

I’m right? I’m missing something? Remember: The first goal is to process input RAW files with silences (block of zeroes in the samples) and output the correct ETI-NI bitstream with the corresponding NULL frames (so the CBR is granted).

JvanKatwijk commented 6 years ago

If you feel you can handle it, then start writing

2018-03-08 12:46 GMT+01:00 lars18th notifications@github.com:

Hi @JvanKatwijk https://github.com/jvankatwijk ,

If you think it is difficult then do not start with a major rewrite

I feel I can handle it! 😄 Regarding the first goal (generation of ETI NULL frames when the incoming data is invalid):

  • After review the "ETI generator" https://github.com/ JvanKatwijk/eti-stuff/blob/master/eti-cmdline/src/eti- handling/eti-generator.cp https://github.com/JvanKatwijk/eti-stuff/blob/master/eti-cmdline/src/eti-handling/eti-generator.cp, I discover that:
    • Initial note: A block (in Mode I) is 32bytes, and 76 blocks are used in one Transmission Frame. The ETI generator consumes blocks from the ring buffer (where the OFDM processor has put the data), and when 76 in total are reached [1 SYNC + 3 FIB + (numberofblocksperCIF*4) CIF; where numberofblocksperCIF=18], then it generates an ETI frame.
    • The reading point is at #L197.
    • The part processing the FIB blocks is at #L216.
    • The part processing the CIF blocks is at #L246.
    • Locations where an error can appear, and the data is discarded:

      • L198: If the "next" block isn't the expected, then the code

        resumes to start (flush the buffer, and waits for the first FIC).

        • When changing to NULL ETI frames, this case needs to be trapped as a discontinuity and process it in consequence, but only reinitialize as a last resort.

      • L209: Block counter overflow.

        • This case here is treated with a simple get-back-counter. I'm not sure if it's the best solution.

      • L267: Continue reading when CIF counters be invalid.

        • This case is uncommon, except when FIB blocks contains invalid data. So, an specific treatment is required.
      • The location when a new ETI frame is created is at #L270.
      • This point is reached when FIC data is completed (FIB blocks completed) and MSC data is completed too (the last CIF block of a Transmission Frame comes).
    • The part when the ETI frame is filled is at #L278.
    • And the location when the ETI frame is written is at #L303.

So assuming that the OFDM processor doesn’t miss packets (that isn’t true now, for exemple when the SYNC packet 0 isn’t detected or SYNC is lost)...

  • etiGenerator::init_eti(): This function has to really fill out NULL frames. So, the needed changes are enable the error level (#L410, partially implemented, but not finished). And handle incorrect internal values (like CIFCounts). However, the first goal is a simple flag for a full NULL frame output when needed.
  • etiGenerator::handle_discontinuity(): This will be a new function for processing the three detected cases when invalid data can be discarded. In fact, it is likely that it will not become a real function, but a block of code.

I’m right? I’m missing something? Remember: The first goal is to process input RAW files with silences (block of zeroes in the samples) and output the correct ETI-NI bitstream with the corresponding NULL frames (so the CBR is granted).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JvanKatwijk/eti-stuff/issues/26#issuecomment-371464723, or mute the thread https://github.com/notifications/unsubscribe-auth/AITzwE_NLAe1Boo_Js1XnKVdMXKIYRqOks5tcRowgaJpZM4SIOoZ .

-- Jan van Katwijk

+31 (0)15 3698980 +31 (0) 628260355