Send Raw Data

[ministerofsillywalks]

Hi! One thing that I've felt was missing from the PICTOR experience was the ability to have the raw data. It's nice to quickly see the results of an observation, but it's hard to do precise analysis of the data. Would it be possible to have an option when you submit the obs. request to receive the raw data file, perhaps in a csv file, for each of the graphs?

[0xCoto]

Hey, thanks for your feedback! I suppose that's possible, but it requires a bit of work from my side (feel free to contribute if you'd really wanna see that - hint: you'll have to add a relevant checkbox option to observe.php and edit observe.py to permit the server (pictortelescope.com) and the client (telescope) to share that parameter).

However, I am more than happy to send you the raw data of your observation, given your Observation ID. If you look at line 120 of observe.py, you'll notice that I keep a copy of every observation's raw data (.dat file), so if a user wishes to process, analyze and inspect their data in greater detail, they have the freedom to do so.

The reason I don't include the .dat in the original email is to avoid confusion, as many users are unfamiliar with Python to run plot.py, nor do they understand how radio astronomy digital signal processing/data analysis works.

But if you think this is worth implementing, it's an easy code change, so I could probably find some time to implement it.

[ministerofsillywalks]

I have an idea about how to do this, but I have a couple of questions about the system:

1. If all the observation parameters are stored and read from last_obs.txt, what is last_obs_duration.txt used for?
2. Can I use a package that has not been used yet? specifically, pandas. Thanks!

[0xCoto]

1. Short answer: user privacy. The telescope fetches last_obs.txt, but the website (observe.php) needs to fetch the last observation request datetime and duration. This was initially used to check if an observation was already running to prevent clogging from observation requests. The reason it doesn’t just fetch last_obs.txt is because that file includes the email address of the observer, which is sensitive data, which is obviously a problem, both ethically and legally (GDPR).

2. For plotting? You can use anything you like of course, but I’d be curious to know what you’d like to play with that plot.py/plot_hi.py and numpy doesn’t already provide.

[ministerofsillywalks]

I want to use pandas to save csv files because it allows you to save with each data array in a labeled column easily, whereas if I just used numpy to write csv's it would be messier. Taking it a step back, I would rather make csv files in addition to the .dat file because the csv's are easier to open. Basically I'm trying to make it as easy as possible to get good data.

P.S. If it's not too big a problem, could you send me the dat file for observation 55275982 so I can test my code? This is one of my own observations. Thanks!

[0xCoto]

You should be able to do this with numpy as well. I've implemented something like this for .txt, but you should be able to adjust it for .csv formatting as well pretty easily, but feel free to use whatever is more convenient for you of course.

Example snippet for saving the calibrated spectrum (you place this in plot_hi.py):

np.savetxt('freq.txt', freq, delimiter='\n', fmt='%1.3f')
np.savetxt('snr.txt', estimate_S_N_simple(spectrum, mask), delimiter='\n', fmt='%1.3f')

I've uploaded your observation.dat file here.

[ministerofsillywalks]

Thanks for the file! I just created the pull request.