search

Nowosad / geopat2

GeoPAT 2 - a suite of modules dedicated to analysis of large datasets in their entirety using spatial and/or temporal patterns
http://sil.uc.edu/index.php?id=geopat2
GNU Lesser General Public License v3.0
45 stars 8 forks source link

No RAM to proceed #38

Closed ghost closed 4 years ago

ghost commented 4 years ago

I am running gpat_pointshis on a small raster tif file with this command:

gpat_pointshis -i s5p_no2_20180701T161259_20180707T175356.tif -o query_signature.txt -s cooc -x 17010906 -y -10113438 -z 20

I am getting No RAM to proceed, is this about computer memory ? if yes, am I using bad parameters as this tif file is very small (~3MB).

Nowosad commented 4 years ago

Hi @bacloud14, what was the size of the raster used to create query_signature.txt?

ghost commented 4 years ago

This is a description from qgis tool,

Emprise -67.3243139128884138,17.9001505510951802 : -65.1908151131045486,18.5649038613436268 Unité degrés Largeur 475 Hauteur 148 Type de Donnée Float64 - nombre à virgule flottante de 64 bits Description du Driver GDAL GTiff Métadonnées du Driver GDAL GeoTIFF Description du jeu de données
Compression Bande 1 DESCRIPTION=NO2_column_number_density STATISTICS_COVARIANCES=1.981022874270826e-11 STATISTICS_MAXIMUM=7.1768918132875e-05 STATISTICS_MEAN=5.7358554251944e-05 STATISTICS_MINIMUM=4.3193063902436e-05 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=4.4508366569822e-06 STATISTICS_VALID_PERCENT=100 Bande 2 DESCRIPTION=tropospheric_NO2_column_number_density STATISTICS_COVARIANCES=1.985800576629695e-11 STATISTICS_MAXIMUM=2.8572676455951e-05 STATISTICS_MEAN=1.3735381679694e-05 STATISTICS_MINIMUM=-4.0542184365222e-07 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=4.45620054431e-06 STATISTICS_VALID_PERCENT=100 Bande 3 DESCRIPTION=stratospheric_NO2_column_number_density STATISTICS_COVARIANCES=4.156065322405524e-14 STATISTICS_MAXIMUM=4.4102442188887e-05 STATISTICS_MEAN=4.3623172553164e-05 STATISTICS_MINIMUM=4.3055618730737e-05 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=2.0386285103988e-07 STATISTICS_VALID_PERCENT=100 Bande 4 DESCRIPTION=NO2_slant_column_number_density STATISTICS_COVARIANCES=1.293863986196858e-10 STATISTICS_MAXIMUM=0.00019621865067165 STATISTICS_MEAN=0.00015446877630782 STATISTICS_MINIMUM=0.00011982057185378 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=1.1374733321286e-05 STATISTICS_VALID_PERCENT=100 Bande 5 DESCRIPTION=tropopause_pressure STATISTICS_COVARIANCES=0.1932624570050397 STATISTICS_MAXIMUM=9582.8482108629 STATISTICS_MEAN=9582.478909903 STATISTICS_MINIMUM=9580.9555840325 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=0.43961626107895 Bande 6 DESCRIPTION=absorbing_aerosol_index STATISTICS_COVARIANCES=0.06369625725356094 STATISTICS_MAXIMUM=1.4019764661789 STATISTICS_MEAN=0.50370142992142 STATISTICS_MINIMUM=-0.076236873865128 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=0.2523811745229 Bande 7 DESCRIPTION=cloud_fraction STATISTICS_COVARIANCES=0.001556044601537503 STATISTICS_MAXIMUM=0.21480551362038 STATISTICS_MEAN=0.043239599074814 STATISTICS_MINIMUM=0 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=0.039446731189511 Bande 8 DESCRIPTION=sensor_altitude STATISTICS_COVARIANCES=596.728867587339 STATISTICS_MAXIMUM=829373.0625 STATISTICS_MEAN=829313.11122426 STATISTICS_MINIMUM=829258.3125 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=24.428034460172 Bande 9 DESCRIPTION=sensor_azimuth_angle STATISTICS_COVARIANCES=0.1536903209201688 STATISTICS_MAXIMUM=69.24584197998 STATISTICS_MEAN=68.526861910679 STATISTICS_MINIMUM=67.831855773926 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=0.39203357116473 Bande 10
DESCRIPTION=sensor_zenith_angle STATISTICS_COVARIANCES=3.519073082171841 STATISTICS_MAXIMUM=62.704952368619 STATISTICS_MEAN=59.257373403412 STATISTICS_MINIMUM=55.619335174561 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=1.8759192632339 Bande 11
DESCRIPTION=solar_azimuth_angle STATISTICS_COVARIANCES=4.138333448118339 STATISTICS_MAXIMUM=-51.56232351227 STATISTICS_MEAN=-56.676932005021 STATISTICS_MINIMUM=-61.139163970947 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=2.0342894209326 Bande 12
DESCRIPTION=solar_zenith_angle STATISTICS_COVARIANCES=0.2350026283363526 STATISTICS_MAXIMUM=10.142203330994 STATISTICS_MEAN=9.1798251607373 STATISTICS_MINIMUM=8.1833152770996 STATISTICS_SKIPFACTORX=1 STATISTICS_SKIPFACTORY=1 STATISTICS_STDDEV=0.48477069665601 Plus d'information
Dimensions X: 475 Y: 148 Bandes: 12 Origine -67.3243,18.5649 Taille du Pixel 0.004491576420597610288,-0.004491576420597610288

In French so sorry for the inconvenience. File are no more than 4MB, Also I run the example on Australian map found here: https://nowosad.github.io/geostat18/geostat18_nowosad#25

Thanks !

Nowosad commented 4 years ago

It would be best if you could share the files and code with me - either here or by email.

ghost commented 4 years ago

I shared some files by mail. I couldn't do so here. Thank you Nowosad/

Nowosad commented 4 years ago

@bacloud14 gpat_pointshisis for categorical rasters only, and gpat_pointsts is for time-series of continuous rasters. What do you want to achieve?

ghost commented 4 years ago

Excuse my ignorance, I am really new to geo processing. Basically, I want to discover spatio-temporal clusters in an unsupervised manner. There is an example using Python here https://annefou.github.io/metos_python/05-scipy/ although this does not treat geo data differently, it is simple k-mean on 2D images. In my case, I want to discover how NO2 flows in the air, so a single image is not sufficient, I want to discover patterns in Raster files along time...

Nowosad commented 4 years ago

@bacloud14 you need to prepare your data first. Each of your files has several bands, and I assume you just need the first one. For this, you can use gdal_translate. The second thing is to add nodata value to your rasters with gdalwarp.

gdal_translate -OF VRT -b 1 s5p_no2_20180707T160011_20180713T174831.tif tmp.vrt
gdalwarp tmp.vrt a1.tif -dstnodata -9999
gdal_translate -OF VRT -b 1 s5p_no2_20180707T174140_20180713T191854.tif tmp.vrt
gdalwarp tmp.vrt a2.tif -dstnodata -9999
gdal_translate -OF VRT -b 1 s5p_no2_20180708T172237_20180714T190743.tif tmp.vrt
gdalwarp tmp.vrt a3.tif -dstnodata -9999
gdal_translate -OF VRT -b 1 s5p_no2_20180709T170334_20180715T184729.tif tmp.vrt
gdalwarp tmp.vrt a4.tif -dstnodata -9999
gdal_translate -OF VRT -b 1 s5p_no2_20180710T164430_20180716T182411.tif tmp.vrt
gdalwarp tmp.vrt a5.tif -dstnodata -9999
gdal_translate -OF VRT -b 1 s5p_no2_20180711T162527_20180718T185658.tif tmp.vrt
gdalwarp tmp.vrt a6.tif -dstnodata -9999
gdal_translate -OF VRT -b 1 s5p_no2_20180712T160623_20180719T105633.tif tmp.vrt
gdalwarp tmp.vrt a7.tif -dstnodata -9999
gdal_translate -OF VRT -b 1 s5p_no2_20180712T174753_20180719T105625.tif tmp.vrt
gdalwarp tmp.vrt a8.tif -dstnodata -9999
gdal_translate -OF VRT -b 1 s5p_no2_20180713T172849_20180719T201407.tif tmp.vrt
gdalwarp tmp.vrt a9.tif -dstnodata -9999

This way you are able to calculate spatial signature of your data:

gpat_gridts -i a1.tif -i a2.tif -i a3.tif -i a4.tif -i a5.tif -i a6.tif -i a7.tif -i a8.tif -i a9.tif -o spatiotemporalgrid -n

And, for exampe, you can calulcate a signature for a specified pair of coordinates:

gpat_pointsts -i spatiotemporalgrid -o stpoints -x -65.5 -y 18

However, if you look closely at your data - you have a lot of missing values. I think you need to fix this problem first, before creating clusters...

ghost commented 4 years ago

I will give it a try. Many thanks !!

ghost commented 4 years ago

Can't I more files as input for gpat_gridts ? Whith 10 files, I got some results, but with 20 files, I have an empty result ..

Nowosad commented 4 years ago

You should have... Are you sure all of the files are correct? Can you again send me an email with the data?