Logarithmic scale & interleaved data

houserockr commented 1 year ago

Hi there,

your lib (almost) made my day, because I've been trying to read data from this website. It contains traces of a frequency response analysis of a guitar effects pedal.

Spoiler alert: this falls in the "help needed" category, I couldn't select this while writing the issue.

I tried to read the following file with your lib and it kinda worked at the end, but I needed to make some changes and I ended up with three major questions. The file: https://heartfx.net/ts9/disk/FRTONMID.DAT

What I changed:

In line 234, I needed to add value 4 to the data_type_decoder dict. No idea what it refers to, unfortunately. I just gave it an arbitrary name and it worked. No biggie.
I needed to comment line 585, where you're trying to incorporate the channel_correction_factor with the np.sqrt(2) division. That didn't work for an entire array of data. Maybe it's even a subsequent error. Also, no biggie imho.

The data:

 {'channel_hdr': [ { 'alias_protected': True,
                     'channel_attribute': 'No attribute',
                     'channel_label': 'Chan  1',
                     'channel_offset': 0.0,
                     'channel_scale': 1.0,
                     'coupling': 'DC',
                     'delay': 0.0,
                     'digital_channel': False,
                     'direction': 'Z',
                     'eng_unit': { 'current': -2,
                                   'factor': 1.0,
                                   'label': 'V',
                                   'length': 4,
                                   'luminal_intensity': 0,
                                   'mass': 2,
                                   'mole': 0,
                                   'plane_angle': 0,
                                   'temperature': 0,
                                   'time': -6},
                     'gate_begin': 0.0,
                     'gate_end': 0.0,
                     'input_impedance': 50.0,
                     'int_2_eng_unit': 1.0,
                     'int_label': 'V',
                     'module_id': 'HP35665A',
                     'offset_unique_record': -1,
                     'overloaded': False,
                     'point_num': 1,
                     'range': -32.94331359863281,
                     'record_size': 192,
                     'serial_number': '3603A03568',
                     'user_delay': 0.0,
                     'weight': 'No weighting',
                     'window': { 'bw': 1.0,
                                 'correction_mode': 'Correction not applied',
                                 'narrow_band_corr': 1.0,
                                 'time_const': 0.0,
                                 'trunc': 0.0,
                                 'wide_band_corr': 1.0,
                                 'window_type': 'Window not applied'}},
                   { 'alias_protected': True,
                     'channel_attribute': 'No attribute',
                     'channel_label': 'Chan  2',
                     'channel_offset': 0.0,
                     'channel_scale': 1.0,
                     'coupling': 'DC',
                     'delay': 0.0,
                     'digital_channel': False,
                     'direction': 'Z',
                     'eng_unit': { 'current': -2,
                                   'factor': 1.0,
                                   'label': 'V',
                                   'length': 4,
                                   'luminal_intensity': 0,
                                   'mass': 2,
                                   'mole': 0,
                                   'plane_angle': 0,
                                   'temperature': 0,
                                   'time': -6},
                     'gate_begin': 0.0,
                     'gate_end': 0.0,
                     'input_impedance': 50.0,
                     'int_2_eng_unit': 1.0,
                     'int_label': 'V',
                     'module_id': 'HP35665A',
                     'offset_unique_record': -1,
                     'overloaded': False,
                     'point_num': 2,
                     'range': -44.94331359863281,
                     'record_size': 192,
                     'serial_number': '3603A03568',
                     'user_delay': 0.0,
                     'weight': 'No weighting',
                     'window': { 'bw': 1.0,
                                 'correction_mode': 'Correction not applied',
                                 'narrow_band_corr': 1.0,
                                 'time_const': 0.0,
                                 'trunc': 0.0,
                                 'wide_band_corr': 1.0,
                                 'window_type': 'Window not applied'}}],
  'data_hdr': [ { 'abscissa_delta_x': 1.0174193661806048,
                  'abscissa_first_x': 20.0,
                  'data_title': 'Freq Resp',
                  'data_type': 'BLAH',
                  'domain': 'Frequency domain',
                  'first_vector_record_num': 0,
                  'last_valid_index': 400,
                  'num_points': 401,
                  'offset_unique_record': -1,
                  'record_size': 134,
                  'scan_data': False,
                  'total_cols': 1,
                  'total_rows': 1,
                  'window_applied': False,
                  'x_data_type': 'double',
                  'x_per_point': 0,
                  'x_resolution_type': 'Logarithmic',
                  'xunit': { 'current': 0,
                             'factor': 6.28318977355957,
                             'label': 'Hz',
                             'length': 0,
                             'luminal_intensity': 0,
                             'mass': 0,
                             'mole': 0,
                             'plane_angle': 2,
                             'temperature': 0,
                             'time': -2},
                  'y_data_type': 'float',
                  'y_is_complex': True,
                  'y_is_normalized': True,
                  'y_is_power_data': True,
                  'y_is_valid': True,
                  'y_per_point': 1,
                  'y_unit_valid': False,
                  'yunit': { 'current': 0,
                             'factor': 1.0,
                             'label': '',
                             'length': 0,
                             'luminal_intensity': 0,
                             'mass': 0,
                             'mole': 0,
                             'plane_angle': 0,
                             'temperature': 0,
                             'time': 0}}],
  'file_hdr': { 'application': 'HP 35665A',
                'application_version': 'A.01.11',
                'measurement_start_datetime': datetime.datetime(2020, 1, 11, 16, 2),
                'num_channel_hdr_records': 2,
                'num_data_hdr_records': 1,
                'num_scan_struct_records': 1,
                'num_unique_records': 1,
                'num_vector_hdr_records': 1,
                'num_xdata_records': 0,
                'offset_channel_record': 358,
                'offset_data_hdr_record': 206,
                'offset_scan_struct_record': 1264,
                'offset_unique_record': 742,
                'offset_vector_record': 340,
                'offset_xdata_record': -1,
                'offset_ydata_record': 1304,
                'record_size': 64,
                'sdf_revision': 2},
  'meas_hdr': { 'average_num': 0,
                'average_type': 'None',
                'block_size': 401,
                'center_freq': 10010.0,
                'detection': 'Sample detection',
                'meas_title': '',
                'meas_type': 'Swept measurement',
                'offset_unique_record': -1,
                'pct_overlap': 0.0,
                'real_time': 'Not continuous',
                'record_size': 140,
                'span_freq': 19980.0,
                'start_freq_index': 0,
                'stop_freq_index': 400,
                'sweep_freq': 0.0,
                'sweep_time': 0.0,
                'video_bw': 0.0,
                'zoom_mode_on': True},
  'scan_struct': { 'last_scan_index': 0,
                   'num_of_scans': 1,
                   'record_size': 40,
                   'scan_type': 'Scan',
                   'scan_unit': { 'current': 0,
                                  'factor': 1.0,
                                  'label': 'count',
                                  'length': 0,
                                  'luminal_intensity': 0,
                                  'mass': 0,
                                  'mole': 0,
                                  'plane_angle': 0,
                                  'temperature': 0,
                                  'time': 0},
                   'scan_var_type': 'Float'},
  'valid_file_identifier': True,
  'vector_hdr': [ { 'channel_power_48x': (48, -48),
                    'channel_record': (1, 0),
                    'offset_unique_record': -1,
                    'record_size': 18}]}
array([-3.43252532e-02,  2.08524466e-01, -3.20427716e-02,  2.12265164e-01,
       -2.96738110e-02,  2.16033757e-01, -2.72166301e-02,  2.19829485e-01,
       -2.46694814e-02,  2.23651603e-01, -2.22094133e-02,  2.27286592e-01,
       -1.96668506e-02,  2.30944499e-01, -1.70402769e-02,  2.34624624e-01,
       -1.43281650e-02,  2.38326237e-01, -1.17002707e-02,  2.41877615e-01,
       -8.99290573e-03,  2.45448172e-01, -6.20471546e-03,  2.49037206e-01,
       -3.33433272e-03,  2.52644062e-01, -4.84856166e-04,  2.56090760e-01,
        2.44255154e-03,  2.59551346e-01,  5.44910040e-03,  2.63025045e-01,
        8.53600912e-03,  2.66511202e-01,  1.14805838e-02,  2.69867390e-01,
        1.44989341e-02,  2.73235708e-01,  1.75921433e-02,  2.76615500e-01,
        2.07612999e-02,  2.80006081e-01,  2.36972868e-02,  2.83145308e-01,
        2.66984180e-02,  2.86293775e-01,  2.97655649e-02,  2.89450973e-01,
        3.28995921e-02,  2.92616367e-01,  3.60664465e-02,  2.95845479e-01,
        3.93021703e-02,  2.99083710e-01,  4.26076837e-02,  3.02330524e-01,
        4.59839143e-02,  3.05585325e-01,  4.90219593e-02,  3.08729589e-01,
        5.21208234e-02,  3.11885148e-01,  5.52813001e-02,  3.15051645e-01,
        5.85041903e-02,  3.18228692e-01,  6.16910607e-02,  3.21264774e-01,
        6.49372786e-02,  3.24308813e-01,  6.82435557e-02,  3.27360421e-01,
        7.16106221e-02,  3.30419213e-01,  7.47947246e-02,  3.33619356e-01,
        7.80376717e-02,  3.36832553e-01,  8.13402236e-02,  3.40058535e-01,
        8.47031325e-02,  3.43297035e-01,  8.77753496e-02,  3.46495599e-01,
        9.09009948e-02,  3.49709630e-01,  9.40807387e-02,  3.52938980e-01,
        9.73152742e-02,  3.56183499e-01,  1.00350231e-01,  3.59427512e-01,
        1.03435695e-01,  3.62689048e-01,  1.06572308e-01,  3.65967959e-01,
        1.09760694e-01,  3.69264126e-01,  1.12946630e-01,  3.72702360e-01,
        1.16185844e-01,  3.76160920e-01,  1.19479023e-01,  3.79639804e-01,
        1.22826867e-01,  3.83138925e-01,  1.25683486e-01,  3.86760116e-01,
        1.28585324e-01,  3.90408754e-01,  1.31532952e-01,  3.94084901e-01,
        1.34526983e-01,  3.97788733e-01,  1.37803003e-01,  4.01631415e-01,
        1.41132742e-01,  4.05502439e-01,  1.44516930e-01,  4.09401804e-01,
        1.47956297e-01,  4.13329661e-01,  1.51155710e-01,  4.17201161e-01,
        1.54404595e-01,  4.21101838e-01,  1.57703608e-01,  4.25031841e-01,
        1.61053404e-01,  4.28991318e-01,  1.64233655e-01,  4.33499426e-01,
        1.67464674e-01,  4.38050419e-01,  1.70747146e-01,  4.42644566e-01,
        1.74081832e-01,  4.47282284e-01,  1.77351385e-01,  4.51972187e-01,
        1.80671945e-01,  4.56707239e-01,  1.84044212e-01,  4.61487770e-01,
        1.87468931e-01,  4.66314197e-01,  1.90909937e-01,  4.71342504e-01,
        1.94404855e-01,  4.76421297e-01,  1.97954446e-01,  4.81551021e-01,
        2.01559499e-01,  4.86732185e-01,  2.05324352e-01,  4.92085248e-01,
        2.09149376e-01,  4.97492969e-01,  2.13035449e-01,  5.02955854e-01,
        2.16983467e-01,  5.08474410e-01,  2.21016034e-01,  5.14302909e-01,
        2.25114226e-01,  5.20194232e-01,  2.29279056e-01,  5.26148975e-01,
        2.33511493e-01,  5.32167852e-01,  2.37724558e-01,  5.38473964e-01,
        2.42006138e-01,  5.44851542e-01,  2.46357247e-01,  5.51301241e-01,
        2.50778973e-01,  5.57823956e-01,  2.55549312e-01,  5.64367712e-01,
        2.60399312e-01,  5.70983231e-01,  2.65330195e-01,  5.77671230e-01,
        2.70343184e-01,  5.84432423e-01,  2.75489539e-01,  5.91484249e-01,
        2.80723006e-01,  5.98616123e-01,  2.86045015e-01,  6.05828941e-01,
        2.91456908e-01,  6.13123477e-01,  2.96948582e-01,  6.20456636e-01,
        3.02532494e-01,  6.27872109e-01,  3.08210015e-01,  6.35370791e-01,
        3.13982695e-01,  6.42953575e-01,  3.20494384e-01,  6.51113451e-01,
        3.27124685e-01,  6.59368932e-01,  3.33875597e-01,  6.67720914e-01,
        3.40749174e-01,  6.76170409e-01,  3.47574264e-01,  6.84136868e-01,
        3.54517549e-01,  6.92187905e-01,  3.61580908e-01,  7.00324237e-01,
        3.68766278e-01,  7.08546579e-01,  3.76641005e-01,  7.17307866e-01,
        3.84660423e-01,  7.26165295e-01,  3.92826974e-01,  7.35119641e-01,
        4.01143104e-01,  7.44171858e-01,  4.09803927e-01,  7.53012419e-01,
        4.18622434e-01,  7.61942208e-01,  4.27601188e-01,  7.70961881e-01,
        4.36742842e-01,  7.80071914e-01,  4.46468472e-01,  7.89304793e-01,
        4.56374466e-01,  7.98626661e-01,  4.66463834e-01,  8.08037996e-01,
        4.76739615e-01,  8.17539275e-01,  4.87648159e-01,  8.26873422e-01,
        4.98759568e-01,  8.36286843e-01,  5.10077119e-01,  8.45779717e-01,
        5.21604180e-01,  8.55352104e-01,  5.33636153e-01,  8.64647985e-01,
        5.45889080e-01,  8.74010384e-01,  5.58366477e-01,  8.83439124e-01,
        5.71071863e-01,  8.92933786e-01,  5.84566116e-01,  9.02093709e-01,
        5.98306775e-01,  9.11301255e-01,  6.12297595e-01,  9.20555651e-01,
        6.26542389e-01,  9.29855943e-01,  6.41317785e-01,  9.38937545e-01,
        6.56359375e-01,  9.48052466e-01,  6.71671093e-01,  9.57199454e-01,
        6.87256992e-01,  9.66377199e-01,  7.03698874e-01,  9.75073874e-01,
        7.20432460e-01,  9.83776450e-01,  7.37461925e-01,  9.92483079e-01,
        7.54791319e-01,  1.00119162e+00,  7.72770703e-01,  1.00915229e+00,
        7.91057229e-01,  1.01708484e+00,  8.09654772e-01,  1.02498662e+00,
        8.28567445e-01,  1.03285468e+00,  8.48239064e-01,  1.03992486e+00,
        8.68233800e-01,  1.04692781e+00,  8.88555467e-01,  1.05386031e+00,
        9.09207821e-01,  1.06071842e+00,  9.30626929e-01,  1.06660414e+00,
        9.52380598e-01,  1.07237685e+00,  9.74472344e-01,  1.07803202e+00,
        9.96905446e-01,  1.08356512e+00,  1.02010536e+00,  1.08784771e+00,
        1.04364371e+00,  1.09196186e+00,  1.06752312e+00,  1.09590209e+00,
        1.09174633e+00,  1.09966290e+00,  1.11638880e+00,  1.10211253e+00,
        1.14135981e+00,  1.10434186e+00,  1.16666126e+00,  1.10634458e+00,
        1.19229507e+00,  1.10811460e+00,  1.21844268e+00,  1.10811913e+00,
        1.24489808e+00,  1.10783768e+00,  1.27166164e+00,  1.10726368e+00,
        1.29873395e+00,  1.10639036e+00,  1.32638323e+00,  1.10374403e+00,
        1.35431480e+00,  1.10074449e+00,  1.38252783e+00,  1.09738445e+00,
        1.41102123e+00,  1.09365654e+00,  1.43990600e+00,  1.08835602e+00,
        1.46904266e+00,  1.08264327e+00,  1.49842894e+00,  1.07651067e+00,
        1.52806246e+00,  1.06995058e+00,  1.55633914e+00,  1.06137764e+00,
        1.58478570e+00,  1.05236852e+00,  1.61339819e+00,  1.04291689e+00,
        1.64217281e+00,  1.03301644e+00,  1.67167199e+00,  1.02057362e+00,
        1.70128179e+00,  1.00761414e+00,  1.73099637e+00,  9.94131744e-01,
        1.76080954e+00,  9.80119944e-01,  1.78903556e+00,  9.63720858e-01,
        1.81726694e+00,  9.46793020e-01,  1.84549642e+00,  9.29331899e-01,
        1.87371659e+00,  9.11333203e-01,  1.90095150e+00,  8.91400158e-01,
        1.92810714e+00,  8.70926261e-01,  1.95517528e+00,  8.49908471e-01,
        1.98214722e+00,  8.28343928e-01,  2.00794005e+00,  8.04440022e-01,
        2.03355193e+00,  7.79989660e-01,  2.05897331e+00,  7.54991829e-01,
        2.08419514e+00,  7.29445636e-01,  2.10709643e+00,  6.99975908e-01,
        2.12964869e+00,  6.69980645e-01,  2.15184188e+00,  6.39462709e-01,
        2.17366529e+00,  6.08424902e-01,  2.19312239e+00,  5.76884508e-01,
        2.21216965e+00,  5.44894099e-01,  2.23079872e+00,  5.12458086e-01,
        2.24900031e+00], dtype='>f4')

The questions:

The data only seems to contain the values of the y-axis. How on earth do I get the corresponding x-tics with the frequency? I mean, sure, the header shows start frequency and the range, but how do I need to iterate? Especially if like in my case, frequency was plotted/traced logarithmically?
How do I iterate over the y-values logarithmically?
I tried to determine the x-tics (frequency) by basically starting at the abscissa_first_x: 20 (Hz) and then dividing the entire frequency range (20Hz - 20kHz) by the number of points (401 in this case) and I'm getting the following plot.

foo

Which absolutely irritates me, because it seems as if there are actually two sets of data kind of interleaved in there. What's happening there? I mean one of the graphs actually looks almost right. That hump shouldn't be at 16k, but at 10k. But that could have something to do with log vs. linear scale. But where's the second graph coming from?

I would appreciate ANY hint on how to correctly interpret this weird SDF format by HP. I can only find tiny pieces of information online, it seems as if this is very closed source. :(

matthewrankin commented 1 year ago

The Standard Data Format (SDF) is defined in Appendix B of the "Standard Data Format Utilities User's Guide" version B.02.01, P/N 5963-1715. Figure B-6 shows the file structure and Table B-1 shows common records. Page B-27 starts the section Reconstructing a Trace, which has information calculating the x-values and y-values for a trace. I think that's your best bet to answer your questions.

Per the SDF Utilities User's Guide, the x-values are calculated as follows, since your x_resolution_type is logarithmic:

x0 = abscissa_firstX x1 = x0 abscissa_deltaX x2 = x0 abscissa_deltaX^2 x3 = x0 abscissa_deltaX^3 . . . xn = x0 abscissa_deltaX^n (where n = num_of_points - 1)

given

abscissa_delta_x = 1.0174193661806048 abscissa_first_x = 20.0

x0 = 20 Hz x1 = 20.34838732 Hz x2 = 20.70284333 Hz x3 = 21.063473742594 Hz x400 = 19999.9999999607 Hz

In line 234, I needed to add value 4 to the data_type_decoder dict. No idea what it refers to, unfortunately. I just gave it an arbitrary name and it worked. No biggie.

I hadn't added all the data_type_decoder values. I'll add 4, which is Frequency response.

I needed to comment line 585, where you're trying to incorporate the channel_correction_factor with the np.sqrt(2) division. That didn't work for an entire array of data. Maybe it's even a subsequent error. Also, no biggie imho.

I was cheating, since I've only read data from an HP 35670A. This section of code needs to be cleaned up.

matthewrankin commented 1 year ago

Looks like your y-data is complex ('y_is_complex': True,), which I believe means the real and imaginary values are interleaved. Assuming any factors/corrections were applied appropriately, it appears that:

y0 = -3.43252532e-02 + j2.08524466e-01 y1 = -3.20427716e-02 + j2.12265164e-01

matthewrankin commented 1 year ago

It also looks like I'm returning the wrong number of y-values when the values are complex.

houserockr commented 1 year ago

Thanks for getting back! The reconstruction of a trace on page B-27 is absolute gold! Thanks for sharing! Finally it makes sense :) I think you're right about the number of points, I'm getting n=400, when in fact I get 200 points each of which being a complex value. Also, thanks for pointing out that my y values are complex, I wouldn't have figured that out on my own. But in that case, it makes total sense to interleave both real and imaginary values like that.

The only thing is, now I don't really know how to interpret the y values. You were right, the trace shows a frequency response of a guitar effects pedal from 20-20kHz. However, I was expecting the y values to be scalar values representing the amplitude of the signal in dB. Can I simply take the real values or do I have to sqrt(real^2 + img^2)?

matthewrankin commented 1 year ago

I updated the code (v0.7.0) to return the y-data as python complex numbers when y_is_complex is true in the header file. You should be able to use abs() to determine the magnitude of the data.

matthewrankin commented 1 year ago

There was an error in handling the complex data, which I believe is now fixed in v0.8.2. I created a sample repository that plots the data for the FRTONMAX.DAT, FRTONMID.DAT, and FRTONMIN.DAT files. The plots match what is shown at Ibanez TS-9 Tube Screamer on the bench, so I believe this issue is resolved and ready to be closed.

houserockr commented 1 year ago

Thank you so much for all the info and the fixes of course. I tested it and it works like a charm! The only line of code I "stole" from your create_plot.py is: y_data = 20 * np.log10(np.abs(sdf_data)) :) I learned a lot about SDF format and HP analyzers.

Here's a piece of rationale why I'm doing this: I wanted to figure out, where the TS effects pedal starts to "cut" low frequencies, i.e. where the graph crosses the zero line in the lows. Seems to be at around 170Hz. That's useful information for guitarists :)

matthewrankin commented 1 year ago

Glad I could help. The sdfascii library is better for it and now supports more than just FFT measurements from an HP35670A.

FYI: I moved the sample repository to https://github.com/matthewrankin/sdfascii-examples

questrail / sdfascii

Logarithmic scale & interleaved data #10