search

COSMIC-SETI / FrontPage

Collection point for documents, specifications, notes...
0 stars 0 forks source link

blade-cli is really slow while performing raster scan #7

Closed savinshynu closed 8 months ago

savinshynu commented 1 year ago

Trying to form a raster beam around voyager source with this command: blade-cli --input-type CI8 --output-type F32 -m B -c 131072 -C 1 -T 70 -N 1 -t ATA /mnt/buf1/voyager2/TCOS0001.sb43831462.eb43831531.60060.2801225463.4.1.BD.C480 voyager_raster_bd.bfr5 outfil_bd_raster/

  1. First of all the raster bfr5 has 70 beams and -T cannot be less than number of beams, otherwise code will fail. Not sure why those 2 numbers are coupled. If T > nbeams, it for eg. 80 it is showing cuda allocation error
  2. When I double the channelization or c. there is a cuda allocation error. how does the channelization affects memory. Anyway we are reading the single channel data across time into memory first and then doing all the FFT right.
  3. The code should not be working for some magic numbers.
  4. If I am reading all guppy raw files. the code takes 12 days to run. Or if I'm taking the first 5 files, it is showing 3 days.
  5. Is this how it suppose to be?
radonnachie commented 1 year ago
  1. The beamformer CUDA kernel is written in a way so as to be performant for many spectra and fewer channels. This was in line with the ATA's use case, and doesn't hold up well for our use case. The kernel's logic requires there to be more time (spectra) than beams. Luigi is aware that this is less than ideal (at least for us) and is working toward a better suited kernel.
  2. Doubling the channelisation rate (without changing anything else) will increase the memory usage downstream of the channeliser proportionally. This is because the fine spectra are proportionally larger (double the channels) while the number of spectra beamformed at a time (-T) hasn't changed.
  3. There will be many combinations of parameters that require too much memory, or break internal kernel constraints. So long as there is an explicit error message for the parameter sets, I don't see how things could be different.
  4. I believe prior to now we've only done rasters on a maser, and only used 1 file, which still took about 10 hours. This seems to match the linear relationship between amount of data in and amount of time to process.
  5. Unfortunately everything seems reasonable to me. BLADE could be faster, and that is now the priority seeing as functionality has been verified.
radonnachie commented 1 year ago

Are you writing out to a local NVMe as well, that may be the only thing one could do to ensure optimal performance.

savinshynu commented 1 year ago

No, I am writing filter bank files to a /mnt/slow/. But would that make a big difference?

savinshynu commented 1 year ago

Writing to NVME is shows 1 day and 16 hours. Run time decreased by half. That's good.

Chenoachem commented 8 months ago

@savinshynu do you still see this as an issue?

savinshynu commented 8 months ago

@Chenoa Tremblay @.***> I think it was fixed a long time ago.

-Savin

On Mon, Feb 5, 2024 at 1:42 PM Chenoa Tremblay @.***> wrote:

@savinshynu https://github.com/savinshynu do you still see this as an issue?

— Reply to this email directly, view it on GitHub https://github.com/COSMIC-SETI/FrontPage/issues/7#issuecomment-1928059381, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALICFT6MQIFDIPBQHT3OIKDYSE72DAVCNFSM6AAAAAAXSIT4POVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRYGA2TSMZYGE . You are receiving this because you were mentioned.Message ID: @.***>