search

lofar-astron / RMextract

extract TEC, vTEC, Earthmagnetic field and Rotation Measures from GPS and WMM data for radio interferometry observations
GNU General Public License v3.0
31 stars 22 forks source link

numpy 1.12.x compatibility error in readTEC #5

Closed apmechev closed 6 years ago

apmechev commented 7 years ago

File "/cvmfs/softdrive.nl/lofar_sw/RMextract/lib/python2.7/site-packages/RMextract-0.1-py2.7-linux-x86_64.egg/RMextract/getIONEX.py", line 68, in readTEC times=np.zeros(NumberOfMaps,dtype='float32'); TypeError: 'float' object cannot be interpreted as an index

numpy.version '1.12.1'

Works with 1.9.3

I recast NumberOfMaps to int and it worked.

Now it crashes on /cvmfs/softdrive.nl/lofar_sw/RMextract/lib/python2.7/site-packages/RMextract-0.1-py2.7-linux-x86_64.egg/RMextract/PosTools.py line 398, but that looks like an AIPS error..

twillis449 commented 7 years ago

We are using exactly the same same sequence of python measures commands in ALBUS, except there we go from AZEL coordinates to RADEC. So far things work for me under Ubuntu 16.04 but the default numpy there is 1.11 and I am using casacore libraries v 2.1.0 What version of casacore and pyrap are you using?

apmechev commented 7 years ago

Casacore is casacore-2.2.0

I briefly searched for the particular error (the one solved with int casting) and it looks like it may be a numpy 1.11->1.12 thing. I'm reinstalling the whole stack to see if the Measure() error persists.

Update: Reinstalling the lofar stack from scratch seemed to work. the casacore unit tests all passed this time (didn't check them last time). As a heads up, I'm using casacore-2.2.0.

mknapp55 commented 7 years ago

I ran into the same problem while trying to run the prefactor target pipeline, but reinstalling the lofar stack (via kern-2 ppa) and using numpy 1.12.1 didn't fix it for me. Any tips?

`ubuntu@ip-100-64-109-221:/data/prefactor/scripts$ transfer_gains_RMextract.py --server ftp://ftp.unibe.ch/aiub/CODE/ --prefix CODG --ionexpath /data/t1data/ionex_L556826/ --storedir /data/t1data/calvals_L556830 /data/t1data/L556826_calpost/L556826_SB200_uv.MS transfer_gains_RMextract.py: command not found ubuntu@ip-100-64-109-221:/data/prefactor/scripts$ python transfer_gains_RMextract.py --server ftp://ftp.unibe.ch/aiub/CODE/ --prefix CODG --ionexpath /data/t1data/ionex_L556826/ --storedir /data/t1data/calvals_L556830 /data/t1data/L556826_calpost/L556826_SB200_uv.MS Working on: ['/data/t1data/L556826_calpost/L556826_SB200_uv.MS'] pyrap will be used to compute positions ('earth_rot', 0) ('timerange ', 0) ('use_azel ', False) Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS: 24 columns, 1315770 rows Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS/FIELD: 10 columns, 1 rows Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS/ANTENNA: 10 columns, 60 rows Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS/ANTENNA: 10 columns, 60 rows getting ionexfile for 4984656542.5 file needed: 2016/CODG3050.16I.Z 14 checking /data/t1data/ionex_L556826/CODG3050.16I True file exists /data/t1data/ionex_L556826/CODG3050.16I.Z Traceback (most recent call last): File "transfer_gains_RMextract.py", line 267, in ionex_server=args.server, ionex_prefix=args.prefix, ionexPath=args.ionexpath) File "transfer_gains_RMextract.py", line 150, in main rmdict = get_COMMONROTATION_vals(msinfo, ionex_server, ionex_prefix, ionexPath) File "transfer_gains_RMextract.py", line 81, in get_COMMONROTATION_vals rmdict = getRM.getRM(MSinfo.msname,server=server,prefix=prefix,ionexPath=ionexPath,timestep=300.) File "/usr/local/lib/python2.7/dist-packages/RMextract-0.1-py2.7-linux-x86_64.egg/RMextract/getRM.py", line 158, in getRM tecinfo=ionex.readTEC(ionexf,use_filter=use_filter) File "/usr/local/lib/python2.7/dist-packages/RMextract-0.1-py2.7-linux-x86_64.egg/RMextract/getIONEX.py", line 68, in readTEC times=np.zeros(NumberOfMaps,dtype='float32'); TypeError: 'float' object cannot be interpreted as an index

ubuntu@ip-100-64-109-221:/data/prefactor/scripts$ python transfer_gains_RMextract.py --ionexpath /data/t1data/ionex_L556826/ --storedir /data/t1data/calvals_L556830 /data/t1data/L556826_calpost/L556826_SB200_uv.MS Working on: ['/data/t1data/L556826_calpost/L556826_SB200_uv.MS'] pyrap will be used to compute positions ('earth_rot', 0) ('timerange ', 0) ('use_azel ', False) Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS: 24 columns, 1315770 rows Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS/FIELD: 10 columns, 1 rows Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS/ANTENNA: 10 columns, 60 rows Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS/ANTENNA: 10 columns, 60 rows getting ionexfile for 4984656542.5 Traceback (most recent call last): File "transfer_gains_RMextract.py", line 267, in ionex_server=args.server, ionex_prefix=args.prefix, ionexPath=args.ionexpath) File "transfer_gains_RMextract.py", line 150, in main rmdict = get_COMMONROTATION_vals(msinfo, ionex_server, ionex_prefix, ionexPath) File "transfer_gains_RMextract.py", line 81, in get_COMMONROTATION_vals rmdict = getRM.getRM(MSinfo.msname,server=server,prefix=prefix,ionexPath=ionexPath,timestep=300.) File "/usr/local/lib/python2.7/dist-packages/RMextract-0.1-py2.7-linux-x86_64.egg/RMextract/getRM.py", line 154, in getRM ionexf=ionex.getIONEXfile(time=date_parms,server=server,prefix=prefix,outpath=ionexPath) File "/usr/local/lib/python2.7/dist-packages/RMextract-0.1-py2.7-linux-x86_64.egg/RMextract/getIONEX.py", line 391, in getIONEXfile if "ftp://213.184.6.172/" in server: TypeError: argument of type 'NoneType' is not iterable`

twillis449 commented 7 years ago

Unfortunately I don't do LOFAR so I'm not much help here :(

maaijke commented 7 years ago

SOrry that must be due to an update I just checked in before I left to Perth last week. I'll check the code tomorrow.

On 04/05/2017 04:16 AM, mknapp55 wrote:

I'm having the same problem, but reinstalling the lofar stack (via kern-2 ppa) and using numpy 1.12.1 didn't fix it for me. Any tips?

|ubuntu@ip-100-64-109-221:/data/prefactor/scripts$ transfer_gains_RMextract.py --server ftp://ftp.unibe.ch/aiub/CODE/ --prefix CODG --ionexpath /data/t1data/ionex_L556826/ --storedir /data/t1data/calvals_L556830 /data/t1data/L556826_calpost/L556826_SB200_uv.MS transfer_gains_RMextract.py: command not found ubuntu@ip-100-64-109-221:/data/prefactor/scripts$ python transfer_gains_RMextract.py --server ftp://ftp.unibe.ch/aiub/CODE/ --prefix CODG --ionexpath /data/t1data/ionex_L556826/ --storedir /data/t1data/calvals_L556830 /data/t1data/L556826_calpost/L556826_SB200_uv.MS Working on: ['/data/t1data/L556826_calpost/L556826_SB200_uv.MS'] pyrap will be used to compute positions ('earth_rot', 0) ('timerange ', 0) ('use_azel ', False) Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS: 24 columns, 1315770 rows Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS/FIELD: 10 columns, 1 rows Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS/ANTENNA: 10 columns, 60 rows Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS/ANTENNA: 10 columns, 60 rows getting ionexfile for 4984656542.5 file needed: 2016/CODG3050.16I.Z 14 checking /data/t1data/ionex_L556826/CODG3050.16I True file exists /data/t1data/ionex_L556826/CODG3050.16I.Z Traceback (most recent call last): File "transfer_gains_RMextract.py", line 267, in ionex_server=args.server, ionex_prefix=args.prefix, ionexPath=args.ionexpath) File "transfer_gains_RMextract.py", line 150, in main rmdict = get_COMMONROTATION_vals(msinfo, ionex_server, ionex_prefix, ionexPath) File "transfer_gains_RMextract.py", line 81, in get_COMMONROTATION_vals rmdict = getRM.getRM(MSinfo.msname,server=server,prefix=prefix,ionexPath=ionexPath,timestep=300.) File "/usr/local/lib/python2.7/dist-packages/RMextract-0.1-py2.7-linux-x86_64.egg/RMextract/getRM.py", line 158, in getRM tecinfo=ionex.readTEC(ionexf,use_filter=use_filter) File "/usr/local/lib/python2.7/dist-packages/RMextract-0.1-py2.7-linux-x86_64.egg/RMextract/getIONEX.py", line 68, in readTEC times=np.zeros(NumberOfMaps,dtype='float32'); TypeError: 'float' object cannot be interpreted as an index ubuntu@ip-100-64-109-221:/data/prefactor/scripts$ python transfer_gains_RMextract.py --ionexpath /data/t1data/ionex_L556826/ --storedir /data/t1data/calvals_L556830 /data/t1data/L556826_calpost/L556826_SB200_uv.MS Working on: ['/data/t1data/L556826_calpost/L556826_SB200_uv.MS'] pyrap will be used to compute positions ('earth_rot', 0) ('timerange ', 0) ('use_azel ', False) Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS: 24 columns, 1315770 rows Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS/FIELD: 10 columns, 1 rows Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS/ANTENNA: 10 columns, 60 rows Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS/ANTENNA: 10 columns, 60 rows getting ionexfile for 4984656542.5 Traceback (most recent call last): File "transfer_gains_RMextract.py", line 267, in ionex_server=args.server, ionex_prefix=args.prefix, ionexPath=args.ionexpath) File "transfer_gains_RMextract.py", line 150, in main rmdict = get_COMMONROTATION_vals(msinfo, ionex_server, ionex_prefix, ionexPath) File "transfer_gains_RMextract.py", line 81, in get_COMMONROTATION_vals rmdict = getRM.getRM(MSinfo.msname,server=server,prefix=prefix,ionexPath=ionexPath,timestep=300.) File "/usr/local/lib/python2.7/dist-packages/RMextract-0.1-py2.7-linux-x86_64.egg/RMextract/getRM.py", line 154, in getRM ionexf=ionex.getIONEXfile(time=date_parms,server=server,prefix=prefix,outpath=ionexPath) File "/usr/local/lib/python2.7/dist-packages/RMextract-0.1-py2.7-linux-x86_64.egg/RMextract/getIONEX.py", line 391, in getIONEXfile if "ftp://213.184.6.172/" in server: TypeError: argument of type 'NoneType' is not iterable|

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/maaijke/RMextract/issues/5#issuecomment-291718428, or mute the thread https://github.com/notifications/unsubscribe-auth/AGuwd0yxEVM--gJegRauO-n7VAQbWdPGks5rsvmPgaJpZM4MxiFU.

apmechev commented 7 years ago

For the numpy 1.12.x to work, I recast NumberOfMaps to int with

NumberOfMaps=int(NumberOfMaps)

I did this right before line 68 in getIONEX.py and it seems to have worked, looks like the newer numpy version can't make an array of length float

(this is RE:mknapp55's comment)

maaijke commented 7 years ago

THat's true, we have been getting warnings about that for quite some time now...I guess just ignoring them wasn't the right way. I'll check if there are anymore indices that need recasting, as well as the other error.

On 04/05/2017 12:18 PM, Alexandar Mechev wrote:

For the numpy 1.12.x to work, I recast NumberOfMaps to int with

NumberOfMaps=int(NumberOfMaps)

I did this right before line 68 in getIONEX.py and it seems to have worked, looks like the newer numpy version can't make an array of length float

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/maaijke/RMextract/issues/5#issuecomment-291818379, or mute the thread https://github.com/notifications/unsubscribe-auth/AGuwd1ltojZkc5xhW2PJS0afXSwdsueEks5rs2qCgaJpZM4MxiFU.

maaijke commented 7 years ago

In your case you have an error because the argument "server" is empty. I assume you have the IONEX data already on disc, otherwise you need to provide a server address to collect the data. I 'll adjust the code to give a more meaningfull errror. The integer casting problem should be solved by now.

On 04/05/2017 04:16 AM, mknapp55 wrote:

I'm having the same problem, but reinstalling the lofar stack (via kern-2 ppa) and using numpy 1.12.1 didn't fix it for me. Any tips?

|ubuntu@ip-100-64-109-221:/data/prefactor/scripts$ transfer_gains_RMextract.py --server ftp://ftp.unibe.ch/aiub/CODE/ --prefix CODG --ionexpath /data/t1data/ionex_L556826/ --storedir /data/t1data/calvals_L556830 /data/t1data/L556826_calpost/L556826_SB200_uv.MS transfer_gains_RMextract.py: command not found ubuntu@ip-100-64-109-221:/data/prefactor/scripts$ python transfer_gains_RMextract.py --server ftp://ftp.unibe.ch/aiub/CODE/ --prefix CODG --ionexpath /data/t1data/ionex_L556826/ --storedir /data/t1data/calvals_L556830 /data/t1data/L556826_calpost/L556826_SB200_uv.MS Working on: ['/data/t1data/L556826_calpost/L556826_SB200_uv.MS'] pyrap will be used to compute positions ('earth_rot', 0) ('timerange ', 0) ('use_azel ', False) Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS: 24 columns, 1315770 rows Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS/FIELD: 10 columns, 1 rows Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS/ANTENNA: 10 columns, 60 rows Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS/ANTENNA: 10 columns, 60 rows getting ionexfile for 4984656542.5 file needed: 2016/CODG3050.16I.Z 14 checking /data/t1data/ionex_L556826/CODG3050.16I True file exists /data/t1data/ionex_L556826/CODG3050.16I.Z Traceback (most recent call last): File "transfer_gains_RMextract.py", line 267, in ionex_server=args.server, ionex_prefix=args.prefix, ionexPath=args.ionexpath) File "transfer_gains_RMextract.py", line 150, in main rmdict = get_COMMONROTATION_vals(msinfo, ionex_server, ionex_prefix, ionexPath) File "transfer_gains_RMextract.py", line 81, in get_COMMONROTATION_vals rmdict = getRM.getRM(MSinfo.msname,server=server,prefix=prefix,ionexPath=ionexPath,timestep=300.) File "/usr/local/lib/python2.7/dist-packages/RMextract-0.1-py2.7-linux-x86_64.egg/RMextract/getRM.py", line 158, in getRM tecinfo=ionex.readTEC(ionexf,use_filter=use_filter) File "/usr/local/lib/python2.7/dist-packages/RMextract-0.1-py2.7-linux-x86_64.egg/RMextract/getIONEX.py", line 68, in readTEC times=np.zeros(NumberOfMaps,dtype='float32'); TypeError: 'float' object cannot be interpreted as an index ubuntu@ip-100-64-109-221:/data/prefactor/scripts$ python transfer_gains_RMextract.py --ionexpath /data/t1data/ionex_L556826/ --storedir /data/t1data/calvals_L556830 /data/t1data/L556826_calpost/L556826_SB200_uv.MS Working on: ['/data/t1data/L556826_calpost/L556826_SB200_uv.MS'] pyrap will be used to compute positions ('earth_rot', 0) ('timerange ', 0) ('use_azel ', False) Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS: 24 columns, 1315770 rows Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS/FIELD: 10 columns, 1 rows Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS/ANTENNA: 10 columns, 60 rows Successful readonly open of default-locked table /data/t1data/L556826_calpost/L556826_SB200_uv.MS/ANTENNA: 10 columns, 60 rows getting ionexfile for 4984656542.5 Traceback (most recent call last): File "transfer_gains_RMextract.py", line 267, in ionex_server=args.server, ionex_prefix=args.prefix, ionexPath=args.ionexpath) File "transfer_gains_RMextract.py", line 150, in main rmdict = get_COMMONROTATION_vals(msinfo, ionex_server, ionex_prefix, ionexPath) File "transfer_gains_RMextract.py", line 81, in get_COMMONROTATION_vals rmdict = getRM.getRM(MSinfo.msname,server=server,prefix=prefix,ionexPath=ionexPath,timestep=300.) File "/usr/local/lib/python2.7/dist-packages/RMextract-0.1-py2.7-linux-x86_64.egg/RMextract/getRM.py", line 154, in getRM ionexf=ionex.getIONEXfile(time=date_parms,server=server,prefix=prefix,outpath=ionexPath) File "/usr/local/lib/python2.7/dist-packages/RMextract-0.1-py2.7-linux-x86_64.egg/RMextract/getIONEX.py", line 391, in getIONEXfile if "ftp://213.184.6.172/" in server: TypeError: argument of type 'NoneType' is not iterable|

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/maaijke/RMextract/issues/5#issuecomment-291718428, or mute the thread https://github.com/notifications/unsubscribe-auth/AGuwd0yxEVM--gJegRauO-n7VAQbWdPGks5rsvmPgaJpZM4MxiFU.

maaijke commented 7 years ago

Both issues should be solved now,