devanshkv / fetch

A set of deep learning models for FRB/RFI binary classification.
GNU General Public License v3.0
40 stars 32 forks source link

candmaker.py not working #8

Closed parulj3795 closed 4 years ago

parulj3795 commented 4 years ago

Hi, can you please advise me on the following issue.

$ candmaker.py --frequency_size 256 --time_size 256 --cand_param_file cands.csv --plot --fout /data/candidates/ multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/home/parul/anaconda3/lib/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, *kwds)) File "/home/parul/anaconda3/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar return list(map(args)) File "/home/parul/anaconda3/lib/python3.7/site-packages/fetch-0.1.8-py3.7.egg/EGG-INFO/scripts/candmaker.py", line 52, in cand2h5 cand.get_chunk() File "/home/parul/anaconda3/lib/python3.7/site-packages/pysigproc-0.40-py3.7.egg/candidate.py", line 233, in get_chunk self.data = self.get_data(nstart=nstart, nsamp=nsamp)[:, 0, :] File "/home/parul/anaconda3/lib/python3.7/site-packages/pysigproc-0.40-py3.7.egg/pysigproc.py", line 151, in get_data dtype=self.dtype).reshape((-1,self.nifs,self.nchans)) File "/home/parul/anaconda3/lib/python3.7/site-packages/pysigproc-0.40-py3.7.egg/pysigproc.py", line 130, in dtype raise RuntimeError('nbits=%d not supported' % self.nbits) RuntimeError: nbits=1 not supported """

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/parul/anaconda3/bin/candmaker.py", line 4, in import('pkg_resources').run_script('fetch==0.1.8', 'candmaker.py') File "/home/parul/anaconda3/lib/python3.7/site-packages/pkg_resources/init.py", line 666, in run_script self.require(requires)[0].run_script(script_name, ns) File "/home/parul/anaconda3/lib/python3.7/site-packages/pkg_resources/init.py", line 1462, in run_script exec(code, namespace, namespace) File "/home/parul/anaconda3/lib/python3.7/site-packages/fetch-0.1.8-py3.7.egg/EGG-INFO/scripts/candmaker.py", line 133, in pool.map(cand2h5, process_list, chunksize=1) File "/home/parul/anaconda3/lib/python3.7/multiprocessing/pool.py", line 268, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/home/parul/anaconda3/lib/python3.7/multiprocessing/pool.py", line 657, in get raise self._value RuntimeError: nbits=1 not supported

devanshkv commented 4 years ago

pysigproc by default does not support nbits < 8. While there is an unpack I have not tested it.

Also, we have never tested fetch on data with less than 8-bit resolution so I cannot guarantee anything.

parulj3795 commented 4 years ago

How can I check the resolution? This was a Parkes filterbank file of FRB010125.

devanshkv commented 4 years ago

You can use sigproc's header command, its under the key Number of bits per sample.

devanshkv commented 4 years ago

It's probably related to the input. I've made some changes, please update and try again. There is also an example included now.

parulj3795 commented 4 years ago

cand_tstart_58523 343749200001_tcand_0 1943000_dm_46 76080_snr_7 75050

I used inject_frb to simulate FRBs. Further, the candidates were searched using Heimdall. After running candmaker.py, I get this image for an RFI. Why does the RFI look like this, whereas the FRBs look all good? This is the case with all the RFI.

devanshkv commented 4 years ago

The pulse width, in this case, is 2048 samples, which means we would downsample the data by a factor of width/2 = 1024.

The total data we take is 2 x (number of samples for dispersion delay) or (widthx256)/2 whichever is larger. Here it would be (widthx256)/2 = 131,072. The tcand is 0.19 which is only 2319 samples, so we median pad it.

This is normal behavior. If a candidate has large width (or DM) and a small tcand it would look like this.