Finer Elevation data

[Louis-Backstrom]

Something I noticed while playing around with the spatial data recently, is that we don't have particularly fine-scale coverage of the elevations - the raster file in the \data folder is fine, but seems to me to be unnecessarily big in that it covers the whole of Australia at a very broad scale, meaning we lose out on detail.

This isn't a massive issue but manifests in some strange ways that I've found so far, namely that it cuts off the top part of Mt Glorious, which lies at just over 700m in Brisbane - the highest part of Brisbane (AFAIK) is the start of Lawton Rd, which according to most maps lies between 700 and 720m elevation:

Would it be possible to replace the current raster DEM with one that is finer-scale (say, 20m grid?) but only covers the relevant survey area (BNE LGA boundary) + a small buffer area?

[jeffreyhanson]

Yeah, that should be fairly straightforward. At the moment, we're using the 90m SRTM DEM. We could swap in the 30m SRTM DEM fairly easily (e.g. download data from here: http://dwtkns.com/srtm30m/) - but this could slow down the process for building the assets.

[Louis-Backstrom]

Right - would it be significantly slower if we only downloaded a small region of the DEM, rather than the entirety of Australia like it is currently?

Alternatively - we could replace it with a vector file of contours? That'd probably entail a fairly large rewrite of the asset building code, but might end up being more efficient and accurate.

[jeffreyhanson]

I'm not sure - I'll have to try it and see it how goes.

Nice idea - but in processing vector data (points, lines, polygons) is slower than raster data in general, so I don't think this will be the solution.

[jeffreyhanson]

I've added in some higher resolution elevation data, could you please try rebuilding the assets with it and see if the run-time is acceptable?

[Louis-Backstrom]

Working on the rebuild now. I note that it's broken the book build - should this resolve if/when the assets build?

[jeffreyhanson]

Yeah, looks like it. The log was saying something about a table missing for a species - but the table was present in the zip file. I'm rerunning the build in case something weird happened when the first build was downloading the assets.

[jeffreyhanson]

It looks like something strange must have happened during the first build because the rebuild worked successfully.

[jeffreyhanson]

Also - thanks for rebuilding the assets - let me know if you run into any issues.

[Louis-Backstrom]

 $ include: logi FALSE

  |..........                                                       |  16%
  |..............                                                   |  21%
  |.................                                                |  26%
  |.....................                                            |  32%
  |........................                                         |  37%
  |...........................                                      |  42%
  |...............................                                  |  47%
  |..................................                               |  53%
  |......................................                           |  58%
  |.........................................                        |  63%
  |............................................                     |  68%
  |................................................                 |  74%
  |...................................................              |  79%
  |.......................................................          |  84%
  |..........................................................       |  89%
  |..............................................................   |  95%
  |.................................................................| 100%

output file: /tmp/Rtmp2duZTm/filef5157e811/surveyor-sheet.knit.md

/usr/bin/pandoc +RTS -K512m -RTS /tmp/Rtmp2duZTm/filef5157e811/surveyor-sheet.utf8.md --to latex --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash-implicit_figures --output /tmp/assets/surveyor-sheets/grid-10550-berrinba-east.tex --template /usr/local/lib/R/site-library/rmarkdown/rmd/latex/default-1.17.0.2.tex --highlight-style tango --pdf-engine pdflatex --include-in-header surveyor-sheet-preamble.tex --variable graphics=yes --variable 'geometry:margin=1in'

Output created: assets/surveyor-sheets/grid-10550-berrinba-east.pdf
There were 50 or more warnings (use warnings() to see the first 50)
starting graphs...
  Anseranas semipalmata
Error in grid.newpage() :
  could not open file 'assets/graphs/Anseranas-semipalmata.png'
Calls: <Anonymous> ... grid.draw.ggplot -> print -> print.ggassemble -> grid.newpage
Execution halted
bba
bba

Seems to have failed overnight, just after it finished making the surveyor sheets. Any ideas? I note that the console log is much more informative than it used to be - which is nice, but I don't know what this particular error message now means.

[jeffreyhanson]

That's odd. It looks like its having trouble saving a graph to the file path assets/graphs/Anseranas-semipalmata.png. Do you have the "graphs" folder located in the "assets" folder?

[Louis-Backstrom]

Hmm...

I've just successfully remade the assets but they don't appear to be any different:

Here's Crested Shrike-tit from the new build:

And from the old website build:

This is a species which has been recorded from Lawton Rd before, i.e. over 700m.

I'm going to try and do a complete rebuild again, see if it sorts itself out. Pushing the "new" assets first though.

[Louis-Backstrom]

As an aside, I can't see the assets_backup.zip file anywhere locally - as per #145 - where does (should) it get saved to?

[Louis-Backstrom]

No change with the second rebuild that I can tell.

Another note - the graphs now build two images - one .jpeg and one .png - is this intended?

[jeffreyhanson]

Hmm, just to double check, did you pull the latest version of the atlas before running this? No, there shouldn't be a .jpg and a .png, there should only be a jpg. I'll fix that in a moment.

[Louis-Backstrom]

assets/graphs/Sterna-hirundo.jpeg 2250x1200 24bit N JFIF  [OK] 240878 --> 216881 bytes (9.96%), optimized.
  Thalasseus bergii
assets/graphs/Thalasseus-bergii.jpeg 2250x1200 24bit N JFIF  [OK] 242302 --> 217844 bytes (10.09%), optimized.
  Thalasseus bengalensis
assets/graphs/Thalasseus-bengalensis.jpeg 2250x1200 24bit N JFIF  [OK] 231253 --> 208198 bytes (9.97%), optimized.
  Phaethon lepturus
  Phaethon rubricauda
  Eudyptula minor
  Thalassarche chlororhynchos
  Thalassarche chrysostoma
  Thalassarche cauta
  Thalassarche melanophris
  Oceanites oceanicus
assets/graphs/Oceanites-oceanicus.jpeg 2250x1200 24bit N JFIF  [OK] 217890 --> 195947 bytes (10.07%), optimized.
  Pelagodroma marina
  Fregetta grallaria
  Fregetta tropica
  Macronectes giganteus
  Fulmarus glacialoides
  Daption capense
  Pterodroma gouldi
assets/graphs/Pterodroma-gouldi.jpeg 2250x1200 24bit N JFIF  [OK] 221181 --> 199664 bytes (9.73%), optimized.
  Pterodroma neglecta
  Pterodroma solandri
  Pterodroma inexpectata
  Pterodroma cervicalis
  Pterodroma cookii
  Pterodroma leucoptera
  Pachyptila turtur
assets/graphs/Pachyptila-turtur.jpeg 2250x1200 24bit N JFIF  [OK] 220098 --> 198240 bytes (9.93%), optimized.
  Pachyptila desolata
Error in ans[!test & ok] <- rep(no, length.out = length(ans))[!test &  :
  replacement has length zero
Calls: <Anonymous> ... expand_range4 -> expand_range -> <Anonymous> -> f -> ifelse
In addition: There were 18 warnings (use warnings() to see them)
Execution halted
bba
bba

Failed again - not really sure why. I deleted the entire repo from my computer and recloned before doing this, removed and repulled the latest image, can't think of what I did wrong. Might just be another one of its mysterious dummy spits that will go away next time I build - I'm going to set it running again overnight but if you have any thoughts that'd be great.

[jeffreyhanson]

Yeah - that's odd. I don't know what went wrong there, please let me know if it persists.

[Louis-Backstrom]

Failed again in exactly the same way last night. I've set it running again while I go to uni this morning just as a wild hope, but I'll try and troubleshoot it properly once I get home this afternoon.

[Louis-Backstrom]

3rd fail in a row, same error, same spot.

Googling yields this, which I don't really understand that much, nor why it would specifically be causing a problem with Pachyptila desolata (Antarctic Prion) - which should have enough data points to not stand out?

[Louis-Backstrom]

Still the same error, despite an update of the dataset as an attempt to maybe fix any problems there might have been. @jeffreyhanson any ideas?

[jeffreyhanson]

After some digging, it appears that Pachyptila desolata (and also Ardenna carneipes) has all of its eBird records recorded after its specified starting year located outside the extent of the elevation raster data (e.g. all their records are in the ocean). As a consequence, when calculating the elevation values for its records, all of its records are associated with missing (NA) values. This is a problem because the species metadata spreadsheet (i.e. atlas_list.xlsx) says that this species is supposed to have an elevation graph created for it based on its records, but there is no non-missing data to use for creating the graph. To address this issue, I will add extra code to convert missing elevation values to zeros. This assumption is valid if (i) we assume that the elevation data covers all terrestrial localities and (ii) we do not care about reporting the elevation of records collected in the ocean.

[jeffreyhanson]

Ok, I've just pushed an update which should hopefully fix this issue. @Louis-Backstrom, could you please pull the the latest version and try remaking the assets again?

[Louis-Backstrom]

Ah of course, that makes sense. I wonder, can we make it such that all marine species (ie seabirds) are automatically ignored with respect to elevation?

Also, won't having only 5 graphs mess the formatting up somewhat? Could we insert an empty chart?

I'll rebuild the assets once I get home tonight - at uni.

[jeffreyhanson]

Yeah I like the idea of suppressing all elevation graphs for seabirds. This isn't something I can do automatically though, and will need to be done by updating the spreadsheet.

It won't mess up the formatting with five graphs per se. IIRC, I've coded it to have 3 graphs on the bottom and 2 on the top - so it won't throw an error but it might look a bit ugly. Another option, would be to suppress another graph to keep an even number of graphs. Perhaps we could suppress the habitat type graph for the seabirds too? To suppress habitat and elevation graphs for a given species, we just need to change their graphs to 3_4_5_6 in the excel spreadsheet.

[Louis-Backstrom]

Yeah, that's worth thinking about. For memory, we have all species currently attributed to either "land" or "mixed" (or whatever the terms we've chosen are), with a third "marine" species as a possibility? Could we change all seabirds (excluding species like Gull-billed Tern, Silver Gull, etc. which are found away from the coast not unusually) to marine instead of mixed to solve this? Would this mess up the distribution maps? (Not that they're particularly informative for seabirds, especially given current data coverage).

Suppressing habitat is worthwhile as well I think, given that most "true" seabirds are only found in non-oceanic habitats when breeding, which AFAIK no species (not even our local terns, maybe Silver Gulls do @dbl3raf ??) does in Brisbane. So we don't really lose out there. Thoughts Richard?

Got home an hour ago and am currently midway through building assets - will update with progress...

[jeffreyhanson]

I like the idea, but I don't think changing these species to marine will help. This would affect which grid cells are used for calculating reporting rates and statistics for the species. It won't change which maps are shown, but would exclude any terrestrial records from atlas maps/graphs/statistics (which would influence the content shown in the maps/graphs). To change the graphs shown for the species, we need to change the values in the graphs column of atlas_list.xlsx.

[Louis-Backstrom]

Right, that makes sense. I'll get onto that in the next week or so (as it's not urgent) and do a rebuild (again...) maybe over the weekend to see what the effect is. On the plus side, the latest build just finished and it worked! Elevation seems to have changed as expected, so I think we're nearly at the stage where we can close this issue at last. Just pushing the surveyor sheets now.

[Louis-Backstrom]

Surveyor sheet push failed midway again last night... Otherwise I think this issue is more or less resolved?

[jeffreyhanson]

Great - thank you very much! Yeah, I still don't have any ideas for fixing the surveyor sheet push stuff sorry.

[jeffreyhanson]

Just to double check, shall we close the issue now?

[Louis-Backstrom]

Yes, I think so - I've just created a new issue for the marine species graphs issue we've mentioned, to remind me to do it / address any potential problems that might throw later.

Closing this now.