search

UCBerkeleySETI / rawspec

6 stars 7 forks source link

complex filterbank output #8

Open siemion opened 3 years ago

siemion commented 3 years ago

Add the ability to output complex filterbank output (complex fourier coefficients) concurrently or in place of Stokes I, XX or YY.

david-macmahon commented 3 years ago

What does this mean? The cross-polarization mode already outputs real(XY) and imag(XY).

siemion commented 3 years ago

This would be to acquire just the individual complex Fourier components after channelization for extracting small parts of the band following candidate identification.

david-macmahon commented 3 years ago

I think this would be better handled by a separate program and certainly a different output format than filterbank which has no provision for indicating A) that the data are voltages or B) that the data are complex. It's not like these "voltage filterbank" files would be usable with any existing software that reads filterbank files, so why shoe horn it into that file format?

jack-h commented 3 years ago

Is there an existing (i.e. used by BL) file format which is a reasonable to use for this? -- is it just .raw ?

siemion commented 3 years ago

This particular output mode would (typically) only be used for interim storage, so wouldn't need to be a fully formed archival format. Note that when I said "complex filterbank" I was using filterbank in the generic sense rather than the specific Filterbank format sense. In the envisioned use, select voltages would be extracted as a snapshot around candidates of interest. What format these should take is a good question - .raw would be a bit heavy, but maybe a version with restricted keywords would work. We wouldn't want metadata to dominate the total size. Another option might be sigmf - https://github.com/gnuradio/SigMF.

siemion commented 3 years ago

Also per the suggestion about a separate program, in normal usage of this mode we would generate both the voltage and detected products simultaneously, and rawspec already does the latter, so having it be able to output the voltages as well would be ideal. Also would save having to do the channelization twice.

david-macmahon commented 3 years ago

Sending incoming 8-bit coarsely channelized voltage data to the GPU for further channelization and then recording the resultant finely channelized 32-bit floating point voltage data would increase the volume and rate of the recorded data by a factor of 4x, which would mean a 4x reduction in bandwidth capacity per node vs recording just the 8-bit coarsely channelized voltages to begin with.

jack-h commented 3 years ago

I assume the intention would be to keep only a small number of the channels(?) But I'm not sure it isn't easier to just write a new tool to extract a sub-coarse channel window from a raw file if there is a small band of interest.

On Tue, 17 Nov 2020, 11:16 pm david-macmahon, notifications@github.com wrote:

Sending incoming 8-bit coarsely channelized voltage data to the GPU further channelization and then recording the resultant finely channelized 32-bit floating point voltage data would increase the volume and rate of the recorded data by a factor of 4x, which would mean a 4x reduction in bandwidth capacity per node vs recording just the 8-bit coarsely channelized voltages to begin with.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/UCBerkeleySETI/rawspec/issues/8#issuecomment-729271387, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKWM7CVVU2ZTSFSFJM5WS3SQL73BANCNFSM4TXVKJZQ .

radonnachie commented 3 years ago

A 3 part solution seems to cover all grounds of this, and may better highlight any misunderstandings in the goal of this issue.

Just rehashing the context, in case there is a misnomer: the current overall setup (of which rawspec is but a part) produces products that are used to determine if any sub-band of the input is of interest. Storing the fine channelisation ("complex fourier coefficients") of the band of interest provides the source data such that any other calculations can be run.

Producing the finely channeled band of interest would require:

  1. selective coarse channel rawspec processing (bandpassing the coarse channel data for computation frugality)
  2. bandpassing fine channelisation outputs (for storage frugality)
  3. raw complex fine channelisation storage (for data fidelity)

....1. is already implemented within rawspec.

I imagine that specifying 2. could lead to automatic calculation of the parameters for 1., i.e. specifying a fine-channel bandpass would always limit the coarse channels processed to the minimum required. 2. would just effect some sort of truncation of the output (bandpass).

I further imagine 3. would be a default-false flag, and rawspec wouldn't worry about whether or not the action is within best practices. Rather the fine-channel-complex-output-flag would just typically be used with coarse and fine channel bandpasses.

If all is agreeable, I think that the above specifies more atomic issues.

jack-h commented 3 years ago

I think what you're describing matches my understanding.

When combined with the ability to process single coarse channels (which already exists) the two features:

  1. raw_spec can output voltages (not powers) in some file format TBC
  2. raw_spec can output only a subset of the fine channels it generates

combined serve the intentions of this PR.

On Wed, 18 Nov 2020 at 18:02, RocketRoss notifications@github.com wrote:

A 3 part solution seems to cover all grounds of this, and may better highlight any misunderstandings in the goal of this issue.

Just rehashing the context, in case there is a misnomer: the current overall setup (of which rawspec is but a part) produces products that are used to determine if any sub-band of the input is of interest. Storing the fine channelisation ("complex fourier coefficients") of the band of interest provides the source data such that any other calculations can be run.

Producing the finely channeled band of interest would require:

1.

selective coarse channel rawspec processing (bandpassing the coarse channel data for computation frugality) 2.

bandpassing fine channelisation outputs (for storage frugality) 3.

raw complex fine channelisation storage (for data fidelity) 4.

is already implemented within rawspec.

I imagine that specifying 2. could lead to automatic calculation of the parameters for 1., i.e. specifying a fine-channel bandpass would always limit the coarse channels processed to the minimum required. 2. would just effect some sort of truncation of the output (bandpass).

I further imagine 3. would be a default-false flag, and rawspec wouldn't worry about whether or not the action is within best practices. Rather the fine-channel-complex-output-flag would just typically be used with coarse and fine channel bandpasses.

If all is agreeable, I think that the above specifies more atomic issues.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/UCBerkeleySETI/rawspec/issues/8#issuecomment-729856513, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKWM7E7KPUWJV2IZ3SHWD3SQQD3VANCNFSM4TXVKJZQ .