djhoese
commented
10 years ago Hi Kathy, This sounds familiar to a problem I was having with my most recent development version (for the new version 1.2 release), but this shouldn't be happening in the older versions of polar2grid. It looks like there may be some fill values or bad data getting through. You can see from the log that when it is calculating the grid locations (ll2cr) it says that the west and south locations are:
[2014-04-21 22:42:02,156] : PID 32296 : DEBUG : polar2grid.ll2cr : ll2cr : Data west longitude: -743.735535
The best way for me to solve this is to look at the data files. I don't have any good guesses as to the exact problem, but given that this user said 1000m files and 1000m and 250m crefl files worked make me think that its in the code that interpolates the 1000m resolution navigation to 250m resolution. Let me know if it is easy enough to get me the data otherwise I will come up with another debug method.
kathys
Dave, I got an email from one of our great users and beta testers. She is a great help to us. We had this email exchange today, which includes one of the log files from a failed execution. Any ideas?
From: Anikó KERN anikoc@nimbus.elte.hu Subject: Re: IMAPP RealEarth Web Mapping Service for GeoTIFF Satellite Image Display Beta Software Available Date: April 21, 2014 at 3:45:33 PM CDT To: Kathy Strabala kathys@ssec.wisc.edu
Dear Kathy,
I'm so sorry for my description :) Yes, during the real-time and automatic processing of the overpasses the process of modis2gtiff fails sometimes, but a few hours later when I generate the Level1b HDF files again with SeaDAS and start to use the Polar2Grid again it works correctly.
I send you now the log file.
Thank you for your help! Aniko
Kathy Strabala írta: Aniko, I am not sure that I completely understand your problem. When you run in real-time polar2grid does not work, but when you go back and reprocess it does work? When the software fails, it leaves behind a .log file with alot of information in it. Can you please send us one of these log files, along with the exact command you using? Thanks, Kathy
On Apr 21, 2014, at 3:19 PM, Anikó KERN anikoc@nimbus.elte.hu wrote: Dear Kathy,
Thank you for your kind attention and the detailed answer :), these are good news.
Would you mind, if I asked something again?
Now the Polar2grid and the WMS is running automatically for our new overpasses (which means that when there are new available Level1b HDF files then GeoTIFF files will be generated and then uploaded). But it happened in some cases that the process failed during the running of the modis2gtiff for daytime overpasses. The message was the following yesterday:
Running modis2gtiff.sh -f a1.14110.1232.250m.hdf… INFO : Using default grid configuration: 'grids.conf' INFO : Loading grid configuration 'grids.conf' INFO : Loading package provided rescale config: 'rescale_configs/rescale_inc.8bit.conf' INFO : Extracting swaths... INFO : Getting data file info... INFO : Creating binary files for latitude and longitude data INFO : Creating binary files for image data INFO : Determining what grids the data fits in... INFO : Running ll2cr for grid wgs84_fit and bands [('visible', '02'), ('visible', '01')] INFO : row and column calculation complete ERROR : Couldn't find fornav output file 'result_geo_250m_nav_visible_02_wgs84_fit.real4.9999999999999999455752309870428160.9999999999999999455752309870428160' WARNING : fornav failed for geo_250m_nav band, grid wgs84_fit, remapping as reflectance WARNING : Cleaning up for this job... ERROR : Removing visible02 because of bad fornav execution ERROR : Removing visible01 because of bad fornav execution ERROR : The last grid job for grid wgs84_fit was removed ERROR : Fornav was not able to complete any remapping for navigation set geo_250m_nav ERROR : Remapping data failed ls: cannot access /home/modis/polar2grid/data/_vis_tif: No such file or directory
/home/modis/polar2grid/data/modis2gtiff_20140420_123200.log
When I realized it, I started the process again, but from the beginning, generating the Level1b HDF files again with the SeaDAS software. Then the Polar2grid was working perfectly. (I have these two datasets (real-time and postprocessed HDF files), and I tested it again.) I found this until now only in cases of Aqua overpasses, but not for all overpasses of course. I use the same SeaDAS software both for the automatic and manual running.
And, one more thing, the same script with the *1000m.hdf (I mean for example: modis2gtiff.sh -f a1.14110.1232.1000m.hdf) does not show this problem. Neither the crefl2gtiff.sh script (eg. in this way: a1.14110.1232.crefl.{1000m,250m}.hdf). These are working fine in every cases.
Do you think it is connected to the real-time generating of the Level1b files, or do I something wrong?
I’m really sorry for my question and to take your time again. Again and always :) And I hope the answer is simple :)
Aniko
Kathy Strabala írta: Hi Aniko, We are getting ready to release the official version of this product, and I realized that I had not answered all of your questions. Nevertheless I believe that it is a great software, you made again something fascinating. I love DBGE as well, I check our overpasses every day in that application, but this new one is really what I missed :) Can we see in the future the level2 products also in this new application? :) We really hope to release a new version of polar2grid in the future that is able to display some of the level 2 products. Only one last thing came into my mind. I used to look at the well known false color images of bands 3-6-7 (where one can easily distinguish the snow cover by the clouds) and the 7-2-1 (which helps to identify areas affected by floods). Is there any possibility to implement them later to a newer version of this software? We also hope that future version of polar2grid will allow users to select the bands they want to use to create RGB images. Finally, we think we fixed the problem you showed us about Aqua passes crossing the 00 UTC date/time line and not being composited together. Once we release the software, you will have to upgrade to a new VM and copying and using new wmsupload and imapp-upload scripts to fix this. I hope the re-installation goes more easily than the original installation. Thanks again, Kathy Kathleen Strabala Assistant Scientist University of Wisconsin - Madison Cooperative Institute for Meteorological Satellite Studies (CIMSS) 1225 West Dayton Street Madison, WI 53706 Phone: (608) 263-8752 Fax: (608) 262-5974 kathy.strabala@ssec.wisc.edu On Mar 30, 2014, at 8:33 AM, Anikó KERN anikoc@nimbus.elte.hu wrote: Dear Kathy,
It might easily happen that I'm the last one who tested your beta software, very sorry for the delay, but we had many technical problems with the virtual machine.
We are still not ready with testing it, but I know that you are waiting for our results, therefore I send you what we learned until now. Our system administrator is still working on this adaptation, but he just wrote me that he had closed himself out from our Linux server remotely :) while he was checking the firewall rules needed to use the port 8001. It means that we can continue to adapt and test the software only tomorrow, sitting down in front of that machine physically.
Do you plan that this software should be installed on any kind of Linux? I understand from the manual that we should have RedHat or CentOS Linux to install the VM, but there are many other Linux distributions out there. Does the other DB stations these systems as well?
We think that the present form of this software is appropriate really only for your systems (RedHat or CentOS). However our administrator is seemed to solve every problem arose until, but as he told us it was not simple at all. He started to overwrite the installation script, but then he had to finish the installation manually, therefore we can not give you any alternative script concerning to the Debian system. In any case, he told me that there are a few files that are located in different directories in our Debian Linux, and there are other tricks that he applied.
Now our VM is working, but we can not still check the port 8001 because of some issues about the firewall (what led to the closing out of every remote user right now :))
The remarks which I started to note during the fist steps of the installation are the followings, but all was concerning the differences originated from the different Linux systems. For example the command ‘yum’ is used (might be only) on Fedora Linux, but on other systems it can be else, eg. on Debian Linux (what we are using) it is apt-get. We had problems with the Libvirtd, because on Debian it has other place and name than it was in the install script. Our administrator modified the script install_realearth-imapp.sh to be compatible with Debian (before he decided to do it manually). We had another problem with Virsh as well, because it turned out that the switch net-update it is not found in our Debian system, therefore our administrator upgraded our full Debian system. There is a question during the installation, the last one: regarding the dataroot (NFS mount point). I think it can not be modified after the installation, but I was not sure which directory should I choose. I think it would be good if it was explained in the documentation before the installation.
Nevertheless I believe that it is a great software, you made again something fascinating. I love DBGE as well, I check our overpasses every day in that application, but this new one is really what I missed :) Can we see in the future the level2 products also in this new application? :)
Only one last thing came into my mind. I used to look at the well known false color images of bands 3-6-7 (where one can easily distinguish the snow cover by the clouds) and the 7-2-1 (which helps to identify areas affected by floods). Is there any possibility to implement them later to a newer version of this software?
As soon as we have the total success with this application, I will write you.
I’m sorry again for my (and our administrator’s) speed :)
Aniko
Kathy Strabala írta: Dear Colleagues, The University of Wisconsin-Madison Space Science and Engineering Center (SSEC) has prepared a software package as part of the International MODIS/AIRS Processing Package (IMAPP) that enables users to display satellite imagery in a Google Maps or Google Earth interface. It is distributed as a virtual machine (VM) that is capable of displaying MODIS (and VIIRS) GeoTIFF images created from the IMAPP/CSPP Polar2grid software package in support of Direct Broadcast users. IMAPP RealEarth is a browser-based GeoTIFF imagery viewer powered by SSEC’s RealEarth Web Mapping System. IMAPP RealEarth allows users to process imagery generated by their IMAPP installation for display in Google Maps, and share that imagery with users through a web mapping service. The following define the host system requirements. Please note: Your system will not be able to support the IMAPP RealEarth Virtual Machine (VM) unless it meets all of these requirements: