marianoguerra
commented
1 year ago 🐸 Spatial Programming 🧑🔬 Deconstructing Datalog ✍️ Stop Drawing Dead Diagrams 🤖 AI Code Search & Authoring
Our Work
📝 Yirgacheffe: trying to do The Right Thing with geospatial data via Michael Dales
As promised in my intro, here’s a little bit of current thinking I wrote up about what feels like I’m going to end up building a DSL to let ecologists work on large datasets.
Currently it’s a Python library that lets ecologists just work with large geospatial files as if they’re variables, like numpy does, but manages the memory side of things, as these files quickly can cause you to run out of memory even on a 1TB RAM machine like we have in the group.
The next thing I have in the wings is trying to hide Python multiprocessing support behind my library, for two reasons:
- I’m already chunking up the computation, so throwing it out to many cores to get concurrency seems like a natural fit. Early on in my time here I tried to encourage the ecologists to think in terms of small programs that would then be run concurrently, but that didn’t really work for how they think of programming, so I think letting them think single threaded whilst I hide concurrency is probably easiest.
- GDAL, the library used for a bunch of the actual geospatial transforms is leaky as anything (or rather, it’s Python bindings are), and so pushing that into child processes is a win, as it’s the only real way to clean up properly.
But at this point I think what I’m not doing is a good fit for Python anymore, so I kind of envisage a Go backend, where I handle the concurrency side of things, and a small front end language where I let the ecologists reason about geospatial files as opaque blobs, and possibly have CSV as a natural thing.
But I also think this is a good fit for a visual programming language, where you connect the CSVs and geospatial files by the kind of operators you’d do normally, and then having that be the ecologist view on the world.
I guess my main aim is to try and not do everything though - if you imagine this was wildly successful, then all I’m doing is making yet another data-processing system that is specialised in one thing that someone will then want to do what I’ve done for some other metric down the line. So I think I want to deliberately keep this focussed/niche rather than accidentally drift into building something generic that will inevitably not be good for other purposes.
[x] Change Month and Week Number
[x] focWeekExport "2022-01-19" "2022-01-26"
[x] Update Search Index
[x] Download New Attachments
[x] Update links
[x] Check that comment links work (push weekly dump with channel summaries)
[x] Check to mention right person for moved messages
[x] Summary
[ ] Hashtags
[x] Set title in newsletter
https://tinyletter.com/
https://tinyletter.com/marianoguerra/letters/
http://localhost:8000/history/
https://marianoguerra.github.io/future-of-coding-weekly/
https://stackedit.io/app#