search

DOI-USGS / ISIS3

Integrated Software for Imagers and Spectrometers v3. ISIS3 is a digital image processing software package to manipulate imagery collected by current and past NASA and International planetary missions.
https://isis.astrogeology.usgs.gov
Other
200 stars 168 forks source link

ISIS Spice Server returning incomplete responses on 4.2.0 #4044

Closed Kelvinrr closed 3 years ago

Kelvinrr commented 4 years ago

ISIS version(s) affected: 4.x.x

Description

When using spiceinit with web=yes, the spice server sometimes gives a partial response.

How to reproduce

Run spiceinit with web=yes, it doesn't seem to matter what cube you use. 

(isis4.2.0) krodriguez@smalls:~$ spiceinit from=I53972019RDR.QUB.cub web=yes
**ERROR** The server sent an unrecognized response [3c73706963655f646174613e0a20203c6170706c69636174696f6e5f6c6f673e0a34373732366637353730323033643230346236353732366536353663373330613230323034653631363936363436373236313664363534333666363436353230323032303230323032303230323032303230323032303230336432303264333533333330333333313061323032303463363536313730353336353633366636653634323032303230323032303230323032303230323032303230323032303230323033643230323436323631373336353266366236353732366536353663373332663663373336623266366536313639363633303330333133323265373436633733306132303230353436313732363736353734343137343734363937343735363436353533363836313730363532303230323032303230323032303364323032343632363137333635326636623635373236653635366337333266373036333662326637303633366233303330333033303339326537343730363330613230323035343631373236373635373435303666373336393734363936663665323032303230323032303230323032303230323032303230336432303238323436323631373336353266366236353732366536353663373332663733373036623266363436353334333333303265363237333730326330613230323032303230323032303230323032303230323032303230323032303230323032303230323032303230323032303230323032303230323032303230323436323631373336353266366236353732366536353663373332663733373036623266366436313732333033393337326536323733373032393061323032303439366537333734373237353664363536653734353036663639366537343639366536373230323032303230323032303230323033643230323832343666363437393733373336353739326636623635373236653635366337333266363336623266366433303331356637333633356636353738373433333338326536323633326330613230323032303230323032303230323032303230323032303230323032303230323032303230323032303230323032303230323032303230323032303230323436663634373937333733363537393266366236353732366536353663373332663636366232663664333033313566373633323339326537343636323930613230323034393665373337343732373536643635366537343230323032303230323032303230323032303230323032303230323032303230336432303465373536633663306132303230353337303631363336353633373236313636373434333663366636333662323032303230323032303230323032303230323032303364323032343666363437393733373336353739326636623635373236653635366337333266373336333663366232663466353234323331356635333433346334623533343334353534326533303330333233353339326537343733363330613230323034393665373337343732373536643635366537343530366637333639373436393666366532303230323032303230323032303230336432303234366636343739373337333635373932663662363537323665363536633733326637333730366232663664333033313566363537383734333333383265363237333730306132303230343936653733373437323735366436353665373434313634363436353665363437353664323032303230323032303230323032303364323032343666363437393733373336353739326636623635373236653635366337333266363936313662326637343638363536643639373334313634363436353665363437353664333033303333326537343639306132303230353336383631373036353464366636343635366332303230323032303230323032303230323032303230323032303230323032303364323032343632363137333635326636343635366437333266366436663663363134643631373237333530366336313665363537343631373237393532363136343639373537333330333033303335326536333735363230613230323034393665373337343732373536643635366537343530366637333639373436393666366535313735363136633639373437393230336432303532363536333666366537333734373237353633373436353634306132303230343936653733373437323735366436353665373435303666363936653734363936653637353137353631366336393734373932303364323035323635363336663665373337343732373536333734363536343061323032303433363136643635373236313536363537323733363936663665323032303230323032303230323032303230323032303230323033643230333230613230323035333666373537323633363532303230323032303230323032303230323032303230323032303230323032303230323032303230336432303639373336393733306134353665363435663437373236663735373030613435366536340a20203c2f6170706c69636174696f6e5f6c6f673e0a20203c6b65726e656c735f6c6162656c3e0a34663632366136353633373432303364323034653631363936363462363537393737366637323634373330613230323034323466343435393566343334663434343532303230323032303230323032303230323032303230323032303230323032303230323032303230323032303230323032303230323032303364323033343339333930613230323034323466343435393334333933393566353234313434343934393230323032303230323032303230323032303230323032303230323032303230323032303230323032303230323032303364323032383333333333393336326533313339326332303333333333393336326533313339326332303333333333373336326533323239306132303230343234663434353935663436353234313464343535663433346634343435323032303230323032303230323032303230323032303230323032303230323032303230323032303230323033643230333133303330333133343061323032303433346334663433346235663435353435663264333533333566333133303337333633373331333133333335333232653330333233353566343334663464353035353534343534343230336432303632333133393339363536363631363433303337333836353632363133343331306132303230343934653533326433353333333033333331356635343532343134653533353832303230323032303230323032303230323032303230323032303230323032303230323032303230323033643230323833303265333032633230333032653330333532633230333032653330323930613230323034393465353332643335333333303333333135663534353234313465353335393230323032303230323032303230323032303230323032303230323032303230323032303230323032303364323032383330326533303263323033303265333032633230333032653330333532393061323032303439346535333264333533333330333333313566343935343532343134653533353332303230323032303230323032303230323032303230323032303230323032303230323032303230336432303238333032653330326332303332333032653330326332303330326533303239306132303230343934653533326433353333333033333331356634393534353234313465353334633230323032303230323032303230323032303230323032303230323032303230323032303230323033643230323833303265333032633230333032653330326332303332333032653330323930613435366536343566346636323661363536333734306130613437373236663735373032303364323034623635373236653635366337333061323032303465363136393636343637323631366436353433366636343635323032303230323032303230323032303230323032303230323033643230326433353333333033333331306132303230346336353631373035333635363336663665363432303230323032303230323032303230323032303230323032303230323032303364323032343632363137333635326636623635373236653635366337333266366337333662326636653631363936363330333033313332326537343663373330613230323035343631373236373635373434313734373436393734373536343635353336383631373036353230323032303230323032303230336432303234363236313733363532663662363537323665363536633733326637303633366232663730363336623330333033303330333932653734373036333061323032303534363137323637363537343530366637333639373436393666366532303230323032303230323032303230323032303230323033643230323835343631363236633635326332303234363236313733363532663662363537323665363536633733326637333730366232663634363533343333333032653632373337303263306132303230323032303230323032303230323032303230323032303230323032303230323032303230323032303230323032303230323032303230323032303234363236313733363532663662363537323665363536633733326637333730366232663664363137323330333933373265363237333730323930613230323034393665373337343732373536643635366537343530366636393665373436393665363732303230323032303230323032303230336432303238353436313632366336353263323032343666363437393733373336353739326636623635373236653635366337333266363336623266366433303331356637333633356636353738373433333338326536323633326330613230323032303230323032303230323032303230323032303230323032303230323032303230323032303230323032303230323032303230323032303230323436663634373937333733363537393266366236353732366536353663373332663636366232663664333033313566373633323339326537343636323930613230323034393665373337343732373536643635366537343230323032303230323032303230323032303230323032303230323032303230336432303465373536633663306132303230353337303631363336353633373236313636373434333663366636333662323032303230323032303230323032303230323032303364323032343666363437393733373336353739326636623635373236653635366337333266373336333663366232663466353234323331356635333433346334623533343334353534326533303330333233353339326537343733363330613230323034393665373337343732373536643635366537].

Note, the large block is the response from the server which is an XML string in HEX form .

Converting it back to ASCII: 

<spice_data>
  <application_log>
47726f7570203d204b65726e656c730a20204e6169664672616d65436f6465202020202020202020202020203d202d35333033310a20204c6561705365636f6e64202020202020202020202020202020203d2024626173652f6b65726e656c732f6c736b2f6e616966303031322e746c730a202054617267657441747469747564655368617065202020202020203d2024626173652f6b65726e656c732f70636b2f70636b30303030392e7470630a2020546172676574506f736974696f6e2020202020202020202020203d202824626173652f6b65726e656c732f73706b2f64653433302e6273702c0a2020202020202020202020202020202020202020202020202020202020202024626173652f6b65726e656c732f73706b2f6d61723039372e627370290a2020496e737472756d656e74506f696e74696e6720202020202020203d2028246f6479737365792f6b65726e656c732f636b2f6d30315f73635f65787433382e62632c0a20202020202020202020202020202020202020202020202020202020202020246f6479737365792f6b65726e656c732f666b2f6d30315f7632392e7466290a2020496e737472756d656e74202020202020202020202020202020203d204e756c6c0a202053706163656372616674436c6f636b20202020202020202020203d20246f6479737365792f6b65726e656c732f73636c6b2f4f5242315f53434c4b534345542e30303235392e7473630a2020496e737472756d656e74506f736974696f6e20202020202020203d20246f6479737365792f6b65726e656c732f73706b2f6d30315f65787433382e6273700a2020496e737472756d656e74416464656e64756d20202020202020203d20246f6479737365792f6b65726e656c732f69616b2f7468656d6973416464656e64756d3030332e74690a202053686170654d6f64656c202020202020202020202020202020203d2024626173652f64656d732f6d6f6c614d617273506c616e6574617279526164697573303030352e6375620a2020496e737472756d656e74506f736974696f6e5175616c697479203d205265636f6e73747275637465640a2020496e737472756d656e74506f696e74696e675175616c697479203d205265636f6e73747275637465640a202043616d65726156657273696f6e202020202020202020202020203d20320a2020536f7572636520202020202020202020202020202020202020203d20697369730a456e645f47726f75700a456e64
  </application_log>
  <kernels_label>
4f626a656374203d204e6169664b6579776f7264730a2020424f44595f434f4445202020202020202020202020202020202020202020202020202020203d203439390a2020424f44593439395f52414449492020202020202020202020202020202020202020202020203d2028333339362e31392c20333339362e31392c20333337362e32290a2020424f44595f4652414d455f434f4445202020202020202020202020202020202020202020203d2031303031340a2020434c4f434b5f45545f2d35335f313037363731313335322e3032355f434f4d5055544544203d20623139396566616430373865626134310a2020494e532d35333033315f5452414e53582020202020202020202020202020202020202020203d2028302e302c20302e30352c20302e30290a2020494e532d35333033315f5452414e53592020202020202020202020202020202020202020203d2028302e302c20302e302c20302e3035290a2020494e532d35333033315f495452414e535320202020202020202020202020202020202020203d2028302e302c2032302e302c20302e30290a2020494e532d35333033315f495452414e534c20202020202020202020202020202020202020203d2028302e302c20302e302c2032302e30290a456e645f4f626a6563740a0a47726f7570203d204b65726e656c730a20204e6169664672616d65436f6465202020202020202020202020203d202d35333033310a20204c6561705365636f6e64202020202020202020202020202020203d2024626173652f6b65726e656c732f6c736b2f6e616966303031322e746c730a202054617267657441747469747564655368617065202020202020203d2024626173652f6b65726e656c732f70636b2f70636b30303030392e7470630a2020546172676574506f736974696f6e2020202020202020202020203d20285461626c652c2024626173652f6b65726e656c732f73706b2f64653433302e6273702c0a2020202020202020202020202020202020202020202020202020202020202024626173652f6b65726e656c732f73706b2f6d61723039372e627370290a2020496e737472756d656e74506f696e74696e6720202020202020203d20285461626c652c20246f6479737365792f6b65726e656c732f636b2f6d30315f73635f65787433382e62632c0a20202020202020202020202020202020202020202020202020202020202020246f6479737365792f6b65726e656c732f666b2f6d30315f7632392e7466290a2020496e737472756d656e74202020202020202020202020202020203d204e756c6c0a202053706163656372616674436c6f636b20202020202020202020203d20246f6479737365792f6b65726e656c732f73636c6b2f4f5242315f53434c4b534345542e30303235392e7473630a2020496e737472756d656e7

The response is expected to be formatted more like: 

<spice_data>
  <application_log>
   . . . 
  </application_log>

   <kernels_label>
   . . . 
  </kernels_label>
  <tables>
    <instrument_pointing>
    . . . 
    </instrument_pointing>

    <instrument_position>
    . . . 
    </instrument_position>

    <body_rotation>
    . . . 
    </body_rotation>

    <sun_position>
    . . . 
    </sun_position>

  </tables>
</spice_data>

Clearly, not all the data is getting across the wire, therefore triggering the unknown response error (which is a catchall parsing error).

This at least occurs when using ISIS 4.2.0 on the spice server.

Possible Solution

Currently, my hunch is this is a server issue as it started when the spice server's ISIS install switched to 4.2.0. Testing the Perl script might give some hints as to what is happening.

Additional context

The spice server is on the internal machine isis-spice-d and the Perl scripts are at isis-spice-d:/var/www/cgi-bin/isis/spiceinit.cgi

ISIS install is at /work1/work2/isis3/isis

/work1/work2/isis3/isis4.2.0 contains a full ISIS4.2.0 install.

jessemapel commented 3 years ago

This issue has been up for a while and we haven't gotten any more posts about this, is this something that is still happening for folks? I've been running spiceinit over the web on Mac with dev builds and it hasn't happened for me, both on and off the DOI VPN.

If this is something you have run into recently please post on here with details about how to reproduce.

jessemapel commented 3 years ago

After restarting the server from #4384, it seems like this is cropping back up. Going to take some time to look into this.

procton commented 3 years ago

I tried updating ISIS from 4.3 to 4.4, but the issue still remains. This is really a blocking issue, please provide some directions to skip it, thanks

jlaura commented 3 years ago

@procton, you can download the spice kernels locally and skip using the SPICE server. See the installation instructions for information about installing the base area and whatever mission(s) you also require.

procton commented 3 years ago

Yes, I am actually doing in this way, but it is an heavy download and the 'on demand' approach was very appreciated by the customer.

jlaura commented 3 years ago

@procton :+1: We absolutely understand and are working to get the server back operating normally. Thank you for your patience and for using the work around in the meantime.

procton commented 3 years ago

The problem seems solved, anyone can confirm the SPICE server is up and reliable ?

jessemapel commented 3 years ago

This was fixed by @acpaquette on issue #4356

acpaquette commented 3 years ago

Unfortunately my change did not fix this issue but the spiceserver should be working fine. We are currently using a 3.5.0 server and not the 4.2.0 server so this should not be an issue for users

procton commented 3 years ago

I am currently using the 4.3 version and it seems fine

acpaquette commented 3 years ago

Yes, but the actual spiceserver that is currently deployed at the USGS is still using version 3.5.0 and the bad response still exists in 4.2.0 and greater versions. The server still works for the majority of ISIS releases that run spiceinit with web=true

bboatwright commented 2 years ago

This issue has cropped again for me in the last day or so. I've been trying to process CTX images and I get **ERROR** The server sent an unrecognized response. after trying to run spiceinit web=yes on the converted image cube.

I'm using the ISIS 3.5.1 build that is installed on Brown University's OSCAR computing cluster: https://docs.ccv.brown.edu/oscar/system-overview

Up until yesterday I'd been running spiceinit with the same specs and having no issues. Likewise, running spiceinit without web=yes works fine. The reason I need web=yes is to pull the kernels for the newest CTX images that aren't included in the local install.

Full print.prt file: print.prt.txt

matthutt commented 2 years ago

I have been running into the same issue since yesterday when processing LRO NAC images with ISIS 5.0.1. Running spiceinit web=true on my cube I get the same error **ERROR** The server sent an unrecognized response..

Here is my print.prt file: print.prt.txt

Kelvinrr commented 2 years ago

It seems the server just started returning empty responses regardless of the POST request sent, probably not an issue with a change in code and maybe something with the server itself.

Looking into it.

Kelvinrr commented 2 years ago

Alright, looks like the issue was server related, after changing some configs and a reset it should be back up.

Kelvinrr commented 2 years ago

@bboatwright @matthutt give it another shot and let me know if you still get the error.

matthutt commented 2 years ago

No more error when I try again now. Thank you @Kelvinrr!

bboatwright commented 2 years ago

Works now for me as well, thanks!

taoyu17 commented 2 years ago

Hi folks, the same issue has just happened to me with LRO NAC images using web=yes. ERROR The server sent an unrecognized response [3c73706963655f646174613e0a20203c61707.......

I have not seen a definite answer but is this a spice server issue which requires just some patience?

Also, does anyone know how large disk space is needed to download the whole LRO kernels? I only have about 10 GB space locally...

Many thanks!

matthutt commented 2 years ago

@taoyu17, I have just tried it and spiceinit web=true is working fine for me with NAC images. I hope that this gets resolved for you, though.

rsync --dry-run --stats tells me the LRO kernels are 260GB (but it says the data for all missions is 793GB, and the ISIS docs say 520GB, so maybe 170GB is more accurate for LRO?). I have not performed this download myself (I am likewise subject to disc space restrictions) so this could be wildly wrong and I would like to know the answer too.

taoyu17 commented 2 years ago

@matthutt Thanks a lot! Yeah spiceinit is now working again this morning so there was a server issue I believe. Thank you so much also for the command to calculate the downloadable size!

harryshewin commented 11 months ago

Hi, I've been having the same issue here. I am using ISIS=8.0.1, trying to process HiRISE images.

Whenever I try to run spiceinit on the converted cube image (web=yes, ckpredicted=yes), I recieve the error message:

ERROR The server sent an unrecognized response.

[Full print.prt attached]

Please let me know if there is something I am needing to do to correct this!

print.prt.pdf

hlethcoe commented 11 months ago

Hello, I am have the same issue as well when processing LRO NAC images. I am using ISIS version 7.0.0 and my command is: spiceinit from= M110296979LE.cub web= true the error I receive is: **ERROR** The server sent an unrecognized response.

My print.prt is attached. print.prt.txt

bboatwright commented 11 months ago

Same here as of yesterday, also with NAC.

Kelvinrr commented 10 months ago

Spice server is down, try the newer URL: https://astrogeology.usgs.gov/apis/ale/v0.9.1/spiceserver/

IT is aware that the default server is down.

hlethcoe commented 10 months ago

Thank you!

Heather

From: Kelvin Rodriguez @.> Sent: Thursday, December 21, 2023 3:13 PM To: DOI-USGS/ISIS3 @.> Cc: Lethcoe, Heather A (US 398N) @.>; Comment @.> Subject: [EXTERNAL] Re: [DOI-USGS/ISIS3] ISIS Spice Server returning incomplete responses on 4.2.0 (#4044)

Spice server is down, try the newer URL: https://astrogeology.usgs.gov/apis/ale/v0.9.1/spiceserver/https://urldefense.us/v3/__https:/astrogeology.usgs.gov/apis/ale/v0.9.1/spiceserver/__;!!PvBDto6Hs4WbVuu7!MqEsipOmBPdlft_-HEHtp3zFvv3Ddhn_QtGKZ023fRqB10CLMneZU88Rmxj1W8WOnJod9-shhCAOxVGyUmlmaZvxbjclUHISo5s$

IT is aware that the default server is down.

— Reply to this email directly, view it on GitHubhttps://urldefense.us/v3/__https:/github.com/DOI-USGS/ISIS3/issues/4044*issuecomment-1867040097__;Iw!!PvBDto6Hs4WbVuu7!MqEsipOmBPdlft_-HEHtp3zFvv3Ddhn_QtGKZ023fRqB10CLMneZU88Rmxj1W8WOnJod9-shhCAOxVGyUmlmaZvxbjclfEeSK7Q$, or unsubscribehttps://urldefense.us/v3/__https:/github.com/notifications/unsubscribe-auth/BERAERRIXW2K5Z5Z3RF6KNLYKS7BLAVCNFSM4R6TJYIKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBWG4YDIMBQHE3Q__;!!PvBDto6Hs4WbVuu7!MqEsipOmBPdlft_-HEHtp3zFvv3Ddhn_QtGKZ023fRqB10CLMneZU88Rmxj1W8WOnJod9-shhCAOxVGyUmlmaZvxbjclUgcLYlM$. You are receiving this because you commented.Message ID: @.**@.>>

Kelvinrr commented 10 months ago

Default server should be back up now.

NatalieGallegos commented 6 months ago

Hello,

I've been having trouble processing NAC images with spiceinit and web=yes, just as above comments. I've attached print.prt. print_prt.txt

appreciate any help I can get.

AustinSanders commented 6 months ago

Hey @NatalieGallegos , we've had a lot of trouble with the spice server going down, and we've migrated the web service. The latest version of ISIS should default to the new server, but you can use the new server in older versions of ISIS by adding url=https://astrogeology.usgs.gov/apis/ale/v0.9.1/spiceserver/ to your spiceinit call.

NatalieGallegos commented 6 months ago

@AustinSanders This works - thank you!

mr-perry commented 3 weeks ago

I apologize for opening this ticket again, but I am running into a similar issue and can't seem to find a way around it.

I've tried changing the url to the new server (as suggested in the documentation and in the above response from @AustinSanders ), but I receive ERROR An error occurred when talking to the server. The server encountered an unexpected error.

spiceinit from=/d0/marstherm/web_output/themis/MT_THEMIS_I80048017_11279/I80048017RDRa.cub url=https://astrogeology.usgs.gov/apis/ale/v0.9.1/spiceserver/ web=True

I am the Systems Manager for MARSTHERM (https://marstherm.psi.edu), which regularly uses ISIS functions for THEMIS processing. We are currently running IsisVersion 3.5.2.0.

I've included my print.prt file for reference.

print.prt.txt

AustinSanders commented 3 weeks ago

Hey @mr-perry, it looks like the web service is up and running; do you mind linking to the data (.cub) you're using so that I can do some testing on my end?

ISIS 3.5.2.0 is considered a legacy version of ISIS, and it is no longer officially supported. Since this is not a supported version, I plan to perform testing on a later version of ISIS (v8.3.0).