Integrating file upload + viewing and naming

[gbathree]

This is relative to use of the library in SurveyStack as a general area drawing survey question type.

Description / Background

In our experience, users may have existing boundaries in standard formats (geojson, kml, wkt, etc.), or they may need to draw some boundaries, or both! They may also want to draw multiple boundaries on a single map because the snapping feature (which thank you! very useful!) helps ensure clean and crisp edges between properties / areas, or they may want to upload some items, show them on the map, and then snap to those edges new areas (@paul121 experienced this w/ RFM).

In cases of creating field boundaries used for purposes outside the farm (ecosystem service markets, property boundaries, etc.) it's really important to be accurate and to not overlap.

In addition, if the items on the map are being pushed to another service (right now we push fields to farmOS via the API, but we may be pushing fields to other services like ESMC or whatever), the other mission critical piece of information is the field name. Field name information is already integrated into many standard formats (geojson, kml, etc.) so it makes a lot of sense to have it integrated into farmOS-map as a standard option.

Features

these are probably not exhaustive... there are surely requirements / dependencies hidden in there, just getting ideas down :)

1. Allow users to optionally upload wkt, kml/kmz, geojson, and show the result on the map. Users should be allowed to both upload a file AND draw if desired in a single experience.
2. Allow users to view the fields in a list view (maybe below the map), show the names, and allow them to edit the names. I put an image below of how this could work (this is a structure we recently implemented for file uploads). It's a little fancy because I'm showing a field outline as the icon, maybe that's overkill but I think it's a really nice UI and I've seen it in other services.
3. Both (1) and (2) above should be designed to be easy to turn off for devs using the library (sorry this may be obvious)... definitely there are situations where adding file upload or area names would confuse users and not be appropriate.

(sorry this is ultra-simple, but you get the idea... the existing farmOS-map is on the top, and the list of fields is below it.)

Funding

It's possible (still in discussion) we have funding for this work. If we don't solve these problems in farmOS-map, we'll need to solve them within SurveyStack anyway. It just seems like this is a chance to improve core library functionality and it may save work across a wider range of applications.

Thoughts?

[mstenta]

Thanks for opening this @gbathree!

Context for others: I suggested this idea on a recent call when we were discussing some of these needs. The core idea was that a farmOS-map behavior (or multiple) could potentially be written to allow for bulk geometry drawing directly in farmOS-map.

We already have a bulk KML import feature in farmOS itself, but this is built in Drupal/PHP, so it can't be used by applications like SurveyStack, which only use farmOS-map.

If we don't solve these problems in farmOS-map, we'll need to solve them within SurveyStack anyway. It just seems like this is a chance to improve core library functionality and it may save work across a wider range of applications.

So on this note... because of the way we've designed farmOS-map, it's actually possible for SurveyStack to sketch this up as a custom behavior for their needs before we consider making it a core behavior included with farmOS-map! Behaviors are a bit like modules/plugins in this regard. You basically just need to add a custom JS file to the same page that you're loading farmOS-map.js, and apply it as a behavior. farmOS itself does this in a few places, for things that it needs specifically, without needing to maintain this code in farmOS-map itself.

I could see these being very useful general behaviors to include in farmOS-map in the future! But let's start by building this as a behavior outside of this repo and then explore a pull request/review process to consider maintaining it here.

(@gbathree I edited your description to replace "farmOS JS" with "farmOS-map" for clarity, because I know that's what you meant. :slightly_smiling_face:)

[mstenta]

Here is the documentation on behaviors, for reference: https://github.com/farmOS/farmOS-map#adding-behaviors

[symbioquine]

I think farmOS-map already exposes most of the functionality needed to do what you're describing @gbathree.

In fact, I've already written a "land drawing tool" for farmOS 2.x which demonstrates something similar (albeit without the import capabilities you've described);

Building on what Mike said - in considering this as a farmOS-map feature request, I think we should look at the following:

1. Does farmOS-map currently present any barriers to implementing "bulk drawing / import" tooling in the consuming code-base (SurveyStack or a farmOS module)? a. Private/awkward APIs? b. Performance limitations/bottlenecks?
2. Are there bits of the functionality that could be pulled into farmOS-map to make this sort of tooling cheaper to write and/or reduce duplicate code/effort between future SurveyStack and core/contrib farmOS implementations?

---

1: These probably need to be found during the process of at least prototyping the tooling for SurveyStack - from that we can learn if there are API/performance problems that are best tackled at the farmOS-map layer.

2: Since core farmOS doesn't have any page right now where you can draw multiple assets or both import and draw at the same time, I suspect there isn't a direct path to making farmOS get these "multi-asset-drawing" or "combined import + drawing" workflows "for free". Given that, it feels like a bit of a "greenfield project" from a farmOS/farmOS-map perspective. My inclination (similar to #⁠1) would be to work backwards from a SurveyStack prototype and say "these bits would be better/cheaper/cleaner/more-robust if implemented directly in farmOS-map".

---

I'd be happy to chat/consult more on this @gbathree :)

[gbathree]

Overall I agree, it'd be great to lay out our thinking / requirements / use cases and ultimately UI first - that'll make the rest of the discussion much easier to have rather than just bullet point ideas :) what you've built is definitely a nice example of a part of what we'd want. In terms of exactly where it all ends up, you're the expert - I'll trust you if it's best to be internal or external to the lib itself.

I think on our side I want to make sure the SoilStack team is open to using this lib in SoilStack, which has totally overlapping feature requirements. If that's the case, I think we have a clear case for improvement and if you're available @symbioquine it'd be great to have you work on it so we could stay on track with other work.

Let me make sure we're all game for this internally and get back to you. Thanks for your thoughtful response!

[paul121]

This is a great initiative! Only one thought from my end, believe this is what @symbioquine is mentioning, but perhaps I can share a specific example.

Allow users to optionally upload wkt, kml/kmz, geojson, and show the result on the map. Users should be allowed to both upload a file AND draw if desired in a single experience.

I worry that adding the logic for these importers within farmOS-map would mean additional dependencies. It would be great if this "import behavior" (included in farmOS-map or separate lib) exposed an API that could be used by the implementing application: SurveyStack pulls in these JS dependencies separately, and farmOS could continue using its existing import logic.

farmOS/Drupal has a serialization/deserialization abstraction that powers the current importer which could be useful for more complex use cases, really whenever the geodata metadata (fields/attributes) comes in a "non-standard" format. @gbathree a good example of this might be the MyFarmKey importer. We need some special logic, but would also be great to leverage a standardized farmOS-map bulk import UI/UX.

But there is an argument for shipping the behavior with all of this too. I guess some initial research would reveal how complex that would be.

[gbathree]

It's a good question about the size and scope of the library. some of these things are pretty easy to convert... others not. I'm wondering if we can support the most efficient ones, and let others be in custom importers.

An external service is ok, but isn't great for offline and of course just offloads the long-term maintenance to someone else :) but maybe you're right... we can't really trust (to the degree useful for general purpose) relatively simple implementations of converters that have low overhead.

We have other applications, like ESMC and I'm sure others going forward, which are just going to need raw flexibility on both ends.

@symbioquine Would you be interested in doing some background work on this, and then perhaps coming to a meeting in 2 weeks on a tuesday at 3p EST? You could talk to our devs and get more of the specifications, and perhaps we can move this along. I'm at gbathree at gm ail if you want to contact me so I can invite you directly.

[symbioquine]

That sounds good @gbathree I'd be up for doing a bit of up-front investigation/prototyping and coming to your meeting - I'll email you shortly.

[gbathree]

So I have a group (ned horning at regen network, working with fibershed group in CA) interested in the streaming field boundaries that maybe could pay (Regen Farmers Mutual has also expressed interest in this). Also, regardless of whether the soilstack folks want to use this library, we definitely do on the surveystack side and I'd like to quote it out.

So I totally recognize here that we haven't fully fleshed these out, so there's scoping work to do, but can you ballpark estimate the following features (probably separately if possible). I'll put some estimates in for scoping and ui/ux feel free to adjust).

Also, I totally understand your other commitments, so if this all takes time (like you're on it for just 10 hours a week or something) that's fine, just give an estimated timeframe for when things could be completed.

1. Complete scoping w/ community (5 hrs, I can help organize this if needed also)
2. UI/UX drawings (7 hrs, I can help with this also if needed)
3. Community feedback, update drawings (5hrs, I can help with this also if needed)
4. Coding work for FarmOS Mapping Library
   • Streaming feature (push button, stream edge of polygon)
   • Import KML / GeoJSON (maybe other formats if possible)
   • Export KML / GeoJSON
   • List areas / rename areas / remove areas feature

one more thing

I know that SoilStack needs offline functionality - specifically pre-caching some mapping areas. Ned Horning from Regen also identified that as a priority, and showed how it's managed in Arc (they allow groups to select areas on a map to pre-cache, and then everyone in that group pre-caches those areas... it's quite nice). We haven't even really discussed this, but wondering if you want to offer any ideas there and perhaps we can get a quote for that as well to pass to Ned and Regen. I also recognize that some or all of this functionality may be outside the scope of the mapping library itself - if you feel that's the case, that's also fine just checking!

Feel free to shoot me an email with those numbers or post here, whatever is more comfortable for you!

[symbioquine]

This is great @gbathree

Estimates aside for a moment, I have a few thoughts/comments on these features;

Streaming feature (push button, stream edge of polygon)

Do we have an existing example of a tool that does this? The reason I ask, is that the code should actually be pretty simple but the location API is a bit finicky in terms of the quality of coordinates it returns on different browsers/OS'. If we don't have an example of this working in the wild, we might want to start with a hacked-up proof-of-concept/prototype just to demonstrate that the quality of the gathered data will be worth making the feature more production-ready.

Import KML / GeoJSON (maybe other formats if possible)

Since we spoke last, I've added rudimentary KML/GeoJSON import to my land drawing tool. I've improved the styling a bit since capturing this GIF, but maybe it can serve as a starting point for discussion of what features/UX you're looking for;

Export KML / GeoJSON

I'll need to understand more about the context/workflow that the export would occur in, but this should be pretty trivial.

Depending what you're trying to do, the existing GeoJSON API endpoints might be a place to start. For example the following URL would return all the location land assets as a GeoJSON feature collection;

https://my-farmos-server.example.com/assets/geojson/full/land?is_location=1

A JS app that's adding polish on top of that could request multiple asset types via the Subrequests module that comes with farmOS;

# the `query` parameter is `encodeURIComponent('[{"uri": "/assets/geojson/full/land?is_location=1", "action": "view"}, {"uri": "/assets/geojson/full/water?is_location=1", "action": "view"}]')`
https://my-farmos-server.example.com/subrequests?_format=json&query=%5B%7B%22uri%22%3A%20%22%2Fassets%2Fgeojson%2Ffull%2Fland%3Fis_location%3D1%22%2C%20%22action%22%3A%20%22view%22%7D%2C%20%7B%22uri%22%3A%20%22%2Fassets%2Fgeojson%2Ffull%2Fwater%3Fis_location%3D1%22%2C%20%22action%22%3A%20%22view%22%7D%5D

List areas / rename areas / remove areas feature

I'll need to understand this one better. I'm assuming you're looking for some capabilities beyond what the land asset views' actions provide.

they allow groups to select areas on a map to pre-cache, and then everyone in that group pre-caches those areas...

I like this idea a lot. I wonder if there's any IP we need to be careful of in copying that feature.

I also recognize that some or all of this functionality may be outside the scope of the mapping library itself - if you feel that's the case, that's also fine just checking!

I suspect we'll find that to be the case for most of this. If it's okay with you though, I'd like to go ahead with the community scoping part of this. I'm happy to do a bit of that exploratory work for free in my capacity as a maintainer/liaison of farmOS-map and maybe we can come back to the scoping/cost estimate part once we hit parts that require coding or super use-case specific mockups/drawings?

[gbathree]

This is great @gbathree

Estimates aside for a moment, I have a few thoughts/comments on these features;

Streaming feature (push button, stream edge of polygon)

Do we have an existing example of a tool that does this? The reason I ask, is that the code should actually be pretty simple but the location API is a bit finicky in terms of the quality of coordinates it returns on different browsers/OS'. If we don't have an example of this working in the wild, we might want to start with a hacked-up proof-of-concept/prototype just to demonstrate that the quality of the gathered data will be worth making the feature more production-ready.

I don't have an example... however... we can look at the existing Arc tool which I believe does it... maybe there's some hints as to how they're doing it. We have had experience in the past in dealing with finicky gps stuff... can suggest some ways we've dealt with it in the UI (basically, showing the # of satellites to the user (or some sense of signal quality) and pre-turning on GPS before hitting the mapping question so it can connect to satellites before which can take several minutes... otherwise, they can be confused as to why the quality is so low for the first few minutes).

Import KML / GeoJSON (maybe other formats if possible)

Since we spoke last, I've added rudimentary KML/GeoJSON import to my land drawing tool. I've improved the styling a bit since capturing this GIF, but maybe it can serve as a starting point for discussion of what features/UX you're looking for;

That looks amazing! I think we have some UX review on that (and again, maybe that piece is not in the mapping lib or maybe we just 'pre-open' that side drawer as we discussed, dunno) to make sure it's clear enough to the users, but I think it's really beautiful.

Export KML / GeoJSON

I'll need to understand more about the context/workflow that the export would occur in, but this should be pretty trivial.

Depending what you're trying to do, the existing GeoJSON API endpoints might be a place to start. For example the following URL would return all the location land assets as a GeoJSON feature collection;

Again - I think this is part of the scoping / community feedback piece to figure out. I don't know if we want to depend on the farmOS API because again this is getting used in instances outside of farmOS (ie where there's no instance) but we can talk through that. I'd say this feature is a bonus less than a must have.

https://my-farmos-server.example.com/assets/geojson/full/land?is_location=1

A JS app that's adding polish on top of that could request multiple asset types via the Subrequests module that comes with farmOS;

# the `query` parameter is `encodeURIComponent('[{"uri": "/assets/geojson/full/land?is_location=1", "action": "view"}, {"uri": "/assets/geojson/full/water?is_location=1", "action": "view"}]')`
https://my-farmos-server.example.com/subrequests?_format=json&query=%5B%7B%22uri%22%3A%20%22%2Fassets%2Fgeojson%2Ffull%2Fland%3Fis_location%3D1%22%2C%20%22action%22%3A%20%22view%22%7D%2C%20%7B%22uri%22%3A%20%22%2Fassets%2Fgeojson%2Ffull%2Fwater%3Fis_location%3D1%22%2C%20%22action%22%3A%20%22view%22%7D%5D

List areas / rename areas / remove areas feature

I'll need to understand this one better. I'm assuming you're looking for some capabilities beyond what the land asset views' actions provide.

I think what you're showing about is 90% of what's needed (again subject to broader community review) but you're really close, I love what you've already built.

they allow groups to select areas on a map to pre-cache, and then everyone in that group pre-caches those areas...

I like this idea a lot. I wonder if there's any IP we need to be careful of in copying that feature.

Hmmm... yeah dunno, hope not (?) but depends who's serving what. Maybe Will/Brian/Joel from our team could provide input as they've also thought about this.

I also recognize that some or all of this functionality may be outside the scope of the mapping library itself - if you feel that's the case, that's also fine just checking!

I suspect we'll find that to be the case for most of this. If it's okay with you though, I'd like to go ahead with the community scoping part of this. I'm happy to do a bit of that exploratory work for free in my capacity as a maintainer/liaison of farmOS-map and maybe we can come back to the scoping/cost estimate part once we hit parts that require coding or super use-case specific mockups/drawings?

Well... so here's the deal. We've got to raise some money to spend the time scoping this well and fairly, and it takes me time to do that and I need some basic numbers just to have the conversation with partners... so if we can parallel tasks (scoping and identifying funding) that's the best. So having even a ballpark with a big plus or and minus is still really helpful, because if possible I'll try to peg the higher end of the range and then we're fairly safe.

I also understand the desire to not overpromise and be accurate... but ballpark hours would be helpful, even big ballparks :), recognizing that things can change and that no one is going to get mad if they do.

Last thing... perhaps we could organize a spec session on these topics, where we get everyone together and really clarify the wants and needs and maybe try to end with some solid mockups that work technically, from a UI/UX perspective, and from a long-term (where are we going in the future) perspective. May be easier than trying to do it offline. I'm happy to help organize that. If we did it sooner than later, it would also help you more accurately estimate time and effort involved.

[symbioquine]

Well... so here's the deal. We've got to raise some money to spend the time scoping this well and fairly, and it takes me time to do that and I need some basic numbers just to have the conversation with partners... so if we can parallel tasks (scoping and identifying funding) that's the best. So having even a ballpark with a big plus or and minus is still really helpful

Ah, okay that makes sense @gbathree! I'll send you an email tomorrow morning (PST) with some ballpark scope numbers and my availability/rate info.

perhaps we could organize a spec session on these topics, where we get everyone together and really clarify the wants and needs and maybe try to end with some solid mockups that work technically, from a UI/UX perspective, and from a long-term (where are we going in the future) perspective. May be easier than trying to do it offline. I'm happy to help organize that. If we did it sooner than later, it would also help you more accurately estimate time and effort involved.

Sounds good. Maybe we can aim for Friday or early next week some time?

[gbathree]

Yeah, I think early next week at best. I'd like to get a pretty diverse crew for that, so it may take a week or two to organize.

Fantastic, thank you! And we totally respect your rate whatever it is so don't take this as a way to pre-set the rate, but just to be transparent and provide backround we bill at around 75 - 85/hr for work like this with direct partners, especially through openteam. If we're doing pass through billing, the difference between actual hourly and billed hourly is put towards core features that no one wants to pay for (security and maintenance and stuff) as well as donation ($5/dev hour goes toward the Black Farmers Fund).

Secretly I'm hoping to have this done before SoilStack does their mapping updates, so then the argument gets a little easier to switch :) :) :)

On Wed, May 18, 2022 at 12:12 PM Symbioquine @.***> wrote:

Well... so here's the deal. We've got to raise some money to spend the time scoping this well and fairly, and it takes me time to do that and I need some basic numbers just to have the conversation with partners... so if we can parallel tasks (scoping and identifying funding) that's the best. So having even a ballpark with a big plus or and minus is still really helpful

Ah, okay that makes sense @gbathree https://github.com/gbathree! I'll send you an email tomorrow morning (PST) with some ballpark scope numbers and my availability/rate info.

perhaps we could organize a spec session on these topics, where we get everyone together and really clarify the wants and needs and maybe try to end with some solid mockups that work technically, from a UI/UX perspective, and from a long-term (where are we going in the future) perspective. May be easier than trying to do it offline. I'm happy to help organize that. If we did it sooner than later, it would also help you more accurately estimate time and effort involved.

Sounds good. Maybe we can aim for Friday or early next week some time?

— Reply to this email directly, view it on GitHub https://github.com/farmOS/farmOS-map/issues/161#issuecomment-1130216651, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANJMIIET2DFKNXIIJVHKLDVKUJHXANCNFSM5UPNIBCA . You are receiving this because you were mentioned.Message ID: @.***>

-- Greg Austic Schedule a 15 https://calendly.com/gbathree/15min / 30 https://calendly.com/gbathree/30-minute-1-1-meeting / 60 https://calendly.com/gbathree/60min min meeting or Check My Calendar https://www.our-sci.net/greg-calendar/ / Availability https://www.our-sci.net/greg-calendar/

+01 919 545 1083 560 Little Lake dr Unit 10 Ann Arbor MI 48103 https://our-sci.net https://bionutrientinstitute.org/

[mstenta]

Exciting to see all these discussions @gbathree and @symbioquine!

I might suggest that we move some of this to a forum topic, since it seems to becoming broader than a single GitHub issue, which should be focused on a single specific task/feature/bug.

FYI we have another issue open already related to "streaming GPS points" here: https://github.com/farmOS/farmOS-map/issues/51

[symbioquine]

Closing in favor of #170