cesium height values not present in iSC instances

[dannymandel]

Per discussion with @qgan7125:

https://mars.cyverse.org/isamples_central/thing/stream?rows=100&fl=producedBy_samplingSite_location_cesium_height It returns the empty result.

https://hyde.cyverse.org/isamples_central/thing/stream?rows=100&fl=producedBy_samplingSite_location_cesium_height hyde gives the same result.

[dannymandel]

Got an email from the folks at Cesium that said this:

You've gone over one or more of your monthly usage quotas for your account with username dannymandel:

Period covered May 01, 2022 to May 31, 2022 Data streaming 156.79 GB of 15.00 GB Bing Maps Sessions 0 of 1000 Geocodes 0 of 50000

[dannymandel]

Looks like something has gone haywire in the Cesium API invocation:

Total rows to calculate = 2217
Total pages to process = 23
Points remaining: 2217
Page 1 loading...
Page 1 computing...
TypeError: Cesium.when is not a function
    at /app/elevate/lib/index.js:22:16
    at new Promise (<anonymous>)
    at Object.elevatePoints (/app/elevate/lib/index.js:12:12)
    at saveResults (/app/elevate/elevate.js:61:39)
    at updateAllRecords (/app/elevate/elevate.js:127:13)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

[dannymandel]

The Cesium API changed and caused us to infinitely loop while asking for the data.

[dannymandel]

Back to the original problem…

Here's an example record (ark:/28722/k2vd6tz9m) that appears to have location data but no height:

http://localhost:8984/solr/#/isb_core_records/query?q=id:%22ark:%2F28722%2Fk2vd6tz9m%22&q.op=OR&indent=true

[dannymandel]

And here's one (ark:/28722/k29g5md2x) that has h3 and height, as well as location data:

http://localhost:8984/solr/#/isb_core_records/query?q=id:%22ark:%2F28722%2Fk29g5md2x%22&q.op=OR&indent=true

[dannymandel]

In fact, if I examine the h3 value in the database for that first one:

isb_1=# select h3 from thing where id='ark:/28722/k2vd6tz9m';
       h3        
-----------------
 8f2c3625918d08d

And then do a solr query based on that h3 value:

http://localhost:8984/solr/#/isb_core_records/query?q=producedBy_samplingSite_location_h3:8f2c3625918d08d&q.op=OR&indent=true

I get 3040 results. And I can see that at least some of them have heights in the records. If I select that Point from our Point cache, I can see that there is a height populated there:

isb_1=# select * from point where h3='8f2c3625918d08d';
       h3        |     longitude      |     latitude      |      height      
-----------------+--------------------+-------------------+------------------
 8f2c3625918d08d | 40.813654185497334 | 37.83063974274093 | 581.583122409521
(1 row)

So there's something about the solr script that is assigning the height value that is missing some records.

[qgan7125]

isamples_central of mars and hyde servers still don't have producedBy_samplingSite_location_cesium_height. I tried to use different start and row. The endpoints always return empty results.

ex. https://hyde.cyverse.org/isamples_central/thing/stream?rows=100&fl=producedBy_samplingSite_location_cesium_h3

https://mars.cyverse.org/isamples_central/thing/stream?rows=1001&start=10000&fl=producedBy_samplingSite_location_cesium_height

[dannymandel]

If you include a filter with records that contain the value, you'll definitely get stuff back (see slack for more details)

e.g.

https://hyde.cyverse.org/isamples_central/thing/stream?rows=1000&start=100000&q=producedBy_samplingSite_location_cesium_height%3A*&fl=producedBy_samplingSite_location_cesium_height,id

[dannymandel]

Even stranger, if I query for all of the records with that particular h3 value:

http://localhost:8984/solr/#/isb_core_records/query?q=producedBy_samplingSite_location_h3:8f2c3625918d08d&q.op=OR&indent=true&fl=id,producedBy_samplingSite_location_h3,producedBy_samplingSite_location_cesium_height&rows=3100

And I search for the record with the missing elevation value (ark:/28722/k2vd6tz9m), I actually see it show up in the list of ids with that elevation value. But if I query the record directly by its id, I don't see producedBy_samplingSite_location_cesium_height or producedBy_samplingSite_location_h3 in the field values.

[dannymandel]

Actually, I see a problem -- the ones with h3 and height populated somehow have that n2t.net prefix still in their records, e.g.

{
        "id":"http://n2t.net/ark:/28722/k2ws8pg06",
        "producedBy_samplingSite_location_h3":["8f2c3625918d08d"],
        "producedBy_samplingSite_location_cesium_height":[581.583122409521]
},

I thought we had purged all of those records but I guess some of them are still lying around.

[dannymandel]

Good news, they aren't in the db:

isb_1=# select count(*) from thing where id like 'http://n2t.net%';
 count 
-------
     0
(1 row)

Bad news, there are some in solr (http://localhost:8984/solr/#/isb_core_records/query?q=id:*n2t.net*&q.op=OR&indent=true&fl=id,producedBy_samplingSite_location_h3,producedBy_samplingSite_location_cesium_height&rows=3100):

"response":{"numFound":159551,"start":0,"numFoundExact":true,"docs":[