New quest: Detail what landuse=orchard produces

[rugk]

What is possible is to ask users for the produce of farmland with perennial plants, a.k.a plantations: apples, pears, bananas, rubber, coconuts, cocoa, coffee, grapes etc. […] This also requires a country-specific list of common produce types, like it is done for the sports quest because otherwise there are too many choices.

Sure, also trees are easy to answer. So we would have to let the user between trees and fruits first:

What is grown on this orchard?

• trees
• fruits
• (more menu:) "This is no orchard" and "Something else" (maybe "vineyard" :smiley:)

So according to the selection we would have to allow the user to select a type of tree and… seems there are only trees(?). Or does OSM categorize bushes into the "trees" category too?

The UI could be similar to the one of the street paving.

Edit: We need images for these crops:

• [x] Edit: tomatoe (later added)
• [x] crop=pepper
• [x] trees=banana_plants
• [x] trees=orange_trees
• [x] trees=lemon_trees|trees=lime_trees
• [x] trees=guava_trees|trees=mango_trees
• [x] trees=coconut_palms
• [x] trees=apple_trees
• [x] landuse=vineyard
• [x] trees=pineapple_plants
• [x] trees=pear_trees
• [x] trees=plum_trees
• [x] trees=coffee_plants
• [x] trees=peach_trees
• [x] trees=grapefruit_trees
• [x] crop=strawberry
• [x] trees=avocado_trees
• [x] trees=apricot_trees
• [x] trees=cherry_trees
• [x] trees=papaya_trees
• [x] trees=cacao_trees
• [x] trees=walnut_trees
• [x] trees=oil_palms
• [x] trees=fig_trees
• [x] trees=tea_plants
• [x] trees=almond_trees
• [x] trees=raspberry
• [x] trees=palms
• [x] trees=piper_plants
• [x] trees=olive_trees
• [x] trees=date_palms
• [x] trees=rubber_trees
• [x] trees=cashew_trees
• [x] trees=hop_plants
• [x] trees=hazel_plants
• [x] trees=chestnut_trees
• [x] trees=blueberry
• [x] crop=sisal
• [x] trees=kiwi_plants
• [x] crop=nut
• [x] trees=pistachio_trees
• [x] trees=nutmeg_trees
• [x] crop=jute
• [x] trees=persimmon_trees
• [x] crop=vanilla
• [x] trees=cranberry
• [x] trees=agave_plants
• [x] trees=kola_trees
• [x] trees=mate_plants
• [x] trees=jojoba_plants

[westnordost]

Forget trees, bushes or fruits. The user (and we) don't care if apples grow on trees or bushes. If he sees apples, he will want to mark "apples" and be done with it. The choices just must be ordered in an order that puts the most expected values (i.e. apples, pears etc for Germany) in front. That is why I mentioned the sports quest because that quest also applies a country-specific order (i.e. cricket goes first in GB). This order needs to be researched and I would say, a good old "world atlas" like we did have them in schools is a good starting point.

[rugk]

The user (and we) don't care if apples grow on trees or bushes.

Oh, this made my day. But you're right…

I am currently looking for statistics. For copyright reasons I do not want/cannot to share one here, so could you drop me a mail, so I can reply?

As for a worldwide overview you could take https://de.statista.com/statistik/daten/studie/28991/umfrage/erzeugung-von-obst-weltweit-nach-arten/ or https://en.wikipedia.org/wiki/Agriculture#Crop_statistics.

[westnordost]

Sent

[rugk]

Ah, I think I finally found a good (and also reliable, for what's it worth) source. It seems you can also download the data as CSV.

--> http://www.fao.org/faostat/en/#rankings/commodities_by_country

One has to tweak it a bit here, it e.g. also includes other things, such as meat, but apart from that, these are some stats.

[rugk]

I've finished my parser, so here it is: https://github.com/rugk/crop-parser The data is here: https://github.com/rugk/crop-parser/blob/master/result/mostPlantedCrops_2014.yml

Now I also know that in Faroe Islands mostly only potatoes :potato: are produced… :smiley:

In the data, we often have many variants of the same "basic" product. E.g. beans in all colours… :wink:

[rugk]

However, one can also see the amount of different fruits there is. (and these are only the top 5 per country) Getting a picture for each one could be difficult/cumbersome, so you likely need to start with a few placeholder pictures.

[westnordost]

Wow, cool. I anticipated you'd go through it by hand. Some notes:

• The final YML should use the OSM tags as defined in the wiki
• Only crops grown on an orchard ("mehrjährig") should be in there (didn't check if this is the case already)
• Seems to be incomplete. I.e. for Germany I am missing i.e. pears, grapes (for wine) as well. So, I would say that the final YML is not something that comes out of your parser but that uses the output of the parser only as a starting point. So, from that perspective, the parser's function was single-use and has now served its term - we got the data from FAO, now we build on top of that

[rugk]

Yeah, converting the crops to OSM tags is another thing, which should be added. However, now you at least have an overview.

Only crops grown on an orchard ("mehrjährig") should be in there (didn't check if this is the case already)

Yeah, there is a blacklist.

Seems to be incomplete.

Ups, yeah. There was a mistake in the parser (it sorted one time too much). This is fixed in https://github.com/rugk/crop-parser/commit/73793baa36d5b3f7f3bffcae4a26f800dd4312c7.

I.e. for Germany I am missing i.e. pears, grapes (for wine) as well.

Grapes are now included. Pears are only in Belgium in the top-5. Seems we overestimate the production… :smile: (at least in tonnes) Taken from the raw data in Germany only 44972 tonnes pears, 155300 pears (dry) and 28917 pears (green) were produced (Note they are not summed up in the parser currently.) . That's less than cabbage ("and other brassica") with 802888t. When we sum up the pears, we still only get 229189, which is still less than a third of the production of cabbage.

[westnordost]

Now it is probably too late to say that, but today it came to me that it may also be a good approach to simply note down the climate zone(s) for each country in an YML and then show the different crops based on these zones. Because I have the impression that what is being planted is more dependent on the climate zone instead of the country. Could this be?

Because, there are just too many crops that can be grown within a country. Even if they are not in the top X, they should still be displayed right? Mentioning all of the possible ones for each and every country is perhaps too much work (to maintain). Perhaps the per-country YML could serve the quest to change the order of the items being displayed and the climate zone determines which crops are shown at all.

[rugk]

Could this be?

Hehe, yeah, maybe. But we would need new data for it. However, island and so on (MH or the example I already mentioned above: Faroe Islands) may be special examples…

Perhaps the per-country YML could serve the quest to change the order of the items being displayed the climate zone determines which crops are shown in the first place.

Hmm, maybe.

[rugk]

So I've looked a bit for statistics and it was quite hard as mostly the natural vegetation is considered, but here at the bottom we have some data, at least for Africa, although the climate zones are of course the similar in the world. (They should be, actually…)

However the data there is still quite vague and the list is often very small. So it is not the data you want to use to limit what should be shown to users. Maybe we need a searchable text input here.

There are also so-called agro-ecological zones, but these have to be made for each region and maybe are even smaller (more detailed) than climate zones, so we won't get far here.

In any case I actually like the hard facts/data we have per country, so maybe sticking to them is the best idea. We can, of course, generate another list of the top-20 crops per country, so we have really every fruit produced there in the list. In a test run I did so and here is the result. We could of course also just include all available data by not limiting the list… (For Germany there are around 70 crops listed)

[westnordost]

Uiuiui, so much data... I think with more than 4 rows of icons in an image list, it really gets unübersichtlich. If there are so many different types of orchard crops, keep in mind that all of the possible ones need to be translated into every language and possibly (if it'll be a list with images) find pictures for it.

So I guess, to limit the work, the quest should only start with the most popular crops and can be extended over time to include more.

The app could ignore those crop types from the YML it does not have resources for.

On 2 July 2017 15:34:15 CEST, rugk notifications@github.com wrote:

So I've looked a bit for statistics and it was quite hard as mostly the natural vegetation is considered, but here at the bottom we have some data, at least for Africa, although the climate zones are of course the similar in the world. (They should be, actually…)

However the data there is still quite vague and the list is often very small. So it is not the data you want to use to limit what should be shown to users. Maybe we need a searchable text input here.

There are also so-called agro-ecological zones, but these have to be made for each region and maybe are even smaller (more detailed) than climate zones, so we won't get far here.

In any case I actually like the hard facts/data we have per country, so maybe sticking to them is the best idea. We can, of course, generate another list of the top-20 crops per country, so we have really every fruit produced there in the list. In a test run I did so and here is the result. We could of course also just include all available data by not limiting the list… (For Germany there are around 70 crops listed)

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/westnordost/StreetComplete/issues/368#issuecomment-312492303

-- Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

[rugk]

Okay, another idea: OSM allows "cereal" as a crop tag? Maybe we could include this? I mean the user may not be able to choose, which cereal exactly and the cereal may change, but at least "cereal" would be something, would not it?

BTW: It's also interesting that strawberries are not in the data. Not at all… I however, know some fields here in Germany. Mostly you can pick them up by yourself (and pay, afterwards). Edit: They are in the data.

[rugk]

Looking through OSM wiki, the tags for crop or produce are really rare. E.g: Are coconuts "crop=nut"?? So our different crops on orchards get less here… :laughing: Bad for OSM, however. They don't even have tomatoes :tomato: there and no soybeans (one of the most planted products of the world)…

I've adjusted the script to convert the output to OSM tags using a small database created out of the (englisch) OSM wiki. So the result when only allowing OSM keys is really much more "cleaned up": https://github.com/rugk/crops-parser/blob/master/result/mostPlantedCrops_2014_OSMonly.yml Due to OSM it, however, is not really a representation of what is mostly planted in a country. YOu can have a look on the other file to see where OSM keys are missing.

PS: As for the climate zone thing I noticed that at least the trees on OSM are actually categorised. (Tropical, Subtropical, Temperate) PPS: Also strange, no cranberry in the source data. Okay, 2014 they maybe were not that popular, but anyway…

[rugk]

OSM or the OSM wiki is somehow bad. Just found some tree tags on another page not even mentioned on the main tree page. Maybe someone can help to fill in the CSV database there.

[westnordost]

I dare to say that by now, you did more research than the one who made the wiki page. You can add new trees if any one tree is not mentioned or documented wrongly

[westnordost]

@CSV database: I'd use trees wherever it makes sense. I have the impression that the trees key is more for plantations/orchards - perennial plants and that the crop key is more for farm landuse.

[rugk]

Yeah, sure I could just add/invent my own tags there, but I want to stay compatible to existing mapping standards. If we assume everyone follows the wiki page I can surely do so…

I'll do some work here…

[westnordost]

The mapping standard is trees=*. Don't expect the wiki to be complete, it is a wiki.

[rugk]

In any case, this question is still open:

Okay, another idea: OSM allows "cereal" as a crop tag? Maybe we could include this? I mean the user may not be able to choose, which cereal exactly and the cereal may change, but at least "cereal" would be something, would not it?

[rugk]

So finally here is a list of missing OSM tags (created with the data from 2013/24, using the blacklist).

[westnordost]

Aren't all of the crops in the list non-permanent (non-perennial) crops?

cereal

Cereals are non perennial, it doesn't make sense to capture this data in OSM when it may change several times a year. Don't suppose that it will be "some kind of cereal" all year through on the same field everywhere. Even in Germany it could be canola ("Raps"), cabbages, beet etc in between, depending on the demand.

Also, only in industrial agriculture with intensive fertilization and pesticides it can be sustained to plant year after year the same crops. Non industrial agriculture will use crop rotation and that would be throughout most of the world.

[rugk]

Ah, yeah, right… Seems sometimes even potatoes are used.

Non industrial agriculture will use crop rotation and that would be throughout most of the world.

(Hopefully) not only there.* I just skimmed the German wiki article on "Fruchtfolgen" and really many things may be used there (potatoes and chick peas…), so we may have to extend our exclusion list even more… That's getting difficult here.

However, some not-so-famous fruits (Ginger, Vanilla, Gooseberries) are likely not used in crop rotation, …

*

Heute ist der Fruchtwechsel ein wichtiger Bestandteil der Landwirtschaft und des modernen Agrarmanagements, sowohl im konventionellen als auch im ökologischen Landbau. […] Es gibt heute weltweit auch eine Konzentration von Monokulturen, zum Beispiel beim Anbau von genverändertem Soja oder Mais in Südamerika und den USA. […] Auch der vereinzelte Anbau von mehrjährigen Selbstfolgen, also der 3- bis 5-malige Anbau gleicher Feldfrüchte, beispielsweise von Mais, gehören heute zur gängigen Praxis

[rugk]

Is not there any farmer with some experience in this field? Not that many on GitHub, probably… :smiley:

I now added some crop OSM tags missing and the list gets huge. And no, it still does not cover any majority of the top-5 as it differs heavily what is planted per country. Also we get this crop rotation problem… So maybe we need to be more radical than using a blacklist there.

My ideas:

1. Use a whitelist?
2. Only include fruits growing on 2.1. trees (farmers likely won't cut down trees to plant different ones that fast) and shrubs 2.2. and lianas, like for kiwis (source) 2.3. and maybe some crops, which can/are only (be) planted in monocultures (rice, source)…

BTW: We also more or less already have a whitelist as the osmcrops.csv, when set to only include OSM tags, acts like a whitelist, which does throw away all crops, where no OSM tag exists.

[rugk]

Actually the definition of an orchard is:

An orchard is an intentional planting of trees or shrubs that is maintained for food production.

However, the article goes on:

A fruit garden is generally synonymous with an orchard, although it is set on a smaller non-commercial scale and may emphasize berry shrubs in preference to fruit trees.

Sure, berries can also be mapped easily…

OSM says:

The tag can also apply to a fruit garden, generally synonymous with an orchard, although it is set on […] (and repeating Wikipedia here)

So we need to include fruit gardens.

The tag cannot be used for a garden producing vegetables under some fruit trees when the vegetable production is the main use and the trees are too scarce.

Good, so no vegetable, nothing growing on ground… (?)

[rugk]

So final question: Should be only include trees :deciduous_tree: ? Or are kiwi :kiwi_fruit: plantages (does such a thing exist?) marked with "orchard" too? And should we maybe drop everything, which may be mapped under "farmland"? So should we drop rice :rice: ?

[westnordost]

Are you asking what would still fall under the OSM orchard tag or are you asking what what should be the scope of this quest?

In any case, we should assume the following: Anything tagged as orchard but without any crop or trees has been tagged from satellite imagery because "it looks like a plantation", ,i.e. systematically planted bushes or trees. This is the input data. This does include kiwis, does include tree nurseries but does not include rice fields.

The user should be able to give a reasonable answer. How his answer is translated into OSM tags, is another thing. I did not concern myself so much with it (did not read OSM wiki), but I would draw the line between orchard (plantation) and farmland exactly between "cultivated on a yearly or less basis" (farmland) and "cultivated permanently / perennial" (orchard). This would put cotton into the farmland category.

[westnordost]

So, basically my answer to using a whitelist is: Anything should be included in the quest that looks like a plantation (from above). Bushes, trees. I.e. I know from survey that these are coconut plantations and these are rubber tree plantations :-].

[westnordost]

But an inclusive list is necessary anyway. After all, the different kinds of plantations all need localization (and a picture).

[rugk]

That's not the easy answer I expected ("just include anything on trees and ignore the rest"), but let's go through it…

Are you asking what would still fall under the OSM orchard tag or are you asking what what should be the scope of this quest?

Both(?)… I mean this quest is about orchards and this quest needs this limit, as we likely do not want to show the question for every other landuse plantage/field/etc. as most of these may probably be about cereals, i.e. about the one big thing, we wanted to exclude here. That's your "low false-positives" rule…

1. "tree nurseries" Unfortunately, we have no data on this. The FAQ. But maybe we need to include this later or find data. For now, I'd exclude them. Also they have yet another key: landuse=forest or landuse=plant_nursery using the produce to specify, what is produced. So maybe this can be a completly different quest. (Maybe open a new issue for it, if you think so…)
2. I get your point with the satellite images, but still rise plantages could be tagged as orchard. If someone just went there, but did not know/see that rise was planted or was too lazy to enter it.
3. Anyway, vegatables, potatos, etc. do not match neither of our criteria (although they seem to be quite different). So I am going to remove them. Cleaning this whole thing up, and hopefully skipping many crops… This work drives one nuts… 😁 But at least I learn something about agriculture. :laughing: If you don't understand this look at this list and this one with crops still TODO…
4. What to do with grapes alá landuse=vineyard?

5. But an inclusive list is necessary anyway.

   Great, we agree… But:

   After all, the different kinds of plantations all need localization (and a picture).

   Good luck! Even the top-5 is so diverse, you won't/would not have imagine(d) it. At least for some small countries or so, you likely need placeholder images. Or text-only or so… But I think this is an issue we can care later about. First we need to have the data filtered and OSM-converted in a useful way.

BTW starting with few fruits and expanding later is not really an option, as the user needs to have the option to select the right thing, when standing in front of a plantation.

[westnordost]

1. & 4.

There is a distinction for which elements the quest is shown and which tags the quest should produce. It should be shown for orchards (= people tagged them as orchards because they areas look like orchards) but the output could be that it is re-tagged as i.e. a vineyard if the crop turns out to be grapes.

I currently have no overview. If it is really that much, perhaps the UI should be a simple input text with autocompletion - no images. (You know, an input text where you start typing to select a predefined value at the end.) But if it is simply that, what to use the ordered list of "likely" crops for then?! D-:

[rugk]

So current status: I've gone through all missing OSM tags. I've included some in the CSV with some suggestions/ideas for OSM keys, and removed some entries in the CSV, which were about foot clearly (= means also visible from the top) not growing in orchards. (such as potatoes or tomatoes) For some controversial crops, I've opened issues in the repo.

The steps going on describes https://github.com/rugk/crops-parser/issues/15.

[westnordost]

This will be the best researched quest of all time.

[rugk]

Oh, yeah… official UN data. What could still be missing? :smile:

[rugk]

@westnordost Could you please comment on the remaining issues? I'd like to get to some decisions there.

BTW I've also added the decision criteria to the wiki there.

[westnordost]

Okay, step 2. Finding photos. I suggest to work by priority: First find photos for the most popular crops, then work down the list. @rugk could you link the priority list here so that if other people want to help know what to search for? The requirements for the photos are:

• 384x384 pixels
• following CC licenses are acceptable: CC 0 (public domain), CC-BY, CC-BY-SA.
• generally: The photos are there to help surveyors to recognize what kind of orchard this is. Usually to display the fruit will be the most helpful thing, but there may be exceptions
• The fruit should be well visible (i.e. fill out most of the space) and it should be a photo of the fruit being on the tree

( I will not help myself but work on other things in the meantime )

[rugk]

@rugk could you link the priority list here so that if other people want to help know what to search for?

Sure, here it is: https://gist.github.com/rugk/7b5dc12efbb0c4625dc79d5c7635ae04

Only includes crops from 2013/14, which have an OSM tag.

Edit: I added this list as a ticklist to the first post, so we can track the current status.

[rugk]

I think we should discuss this image thing in the other repo again. There may be discussions for image selection for each photo, so… better not spam this repo here.

Anybody interested follow https://github.com/rugk/crops-parser/issues, I'll upload some images there. I'm all cropping them to be a square. Making them smaller can be done in a batch conversion later anyway.

I'll later tick the list in this issue here, when the image is okay and all issues about an image are closed. Decision changed: I'll tick everything which is already covered (has an issue or image has been found).

[rugk]

So the images are collected in https://github.com/rugk/crops-parser/tree/master/images. @westnordost if you do not like an image in the dir and no issue is open about the fruit, please open a new one.

[rugk]

Finally I've finished collecting images. There are still some open issues for some fruits (regarding alternative/better pics, etc.), but basically we have pictures for all of them.

So we only need to batch-process them (https://github.com/rugk/crops-parser/issues/35) and then the way is free for implementation.

When implementing, @westnordost, could you provide the translation files/strings (at least for the fruits/crops) early, so I can start translating them. Actually we could even use some automated translation system as it is mostly only one specific term, which has no ambiguity.

[westnordost]

Sure. On Wednesday probably.

Am 24. Juli 2017 16:32:53 MESZ schrieb rugk notifications@github.com:

Finally I've finished collecting images. There are still some open issues for some fruits (regarding alternative/better pics, etc.), but basically we have pictures for all of them.

So we only need to batch-process them (https://github.com/rugk/crops-parser/issues/35) and then the way is free for implementation.

When implementing, @westnordost, could you provide the translation files/strings (at least for the fruits/crops) early, so I can start translating them. Actually we could even use some automated translation system as it is mostly only one specific term, which has no ambiguity.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/westnordost/StreetComplete/issues/368#issuecomment-317441721

-- Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

[westnordost]

I added the strings to POEditor.

[rugk]

Will do translation, at least for German…

[rugk]

POE still shows "100%" translated.

[westnordost]

Someone translated it immediately after I added the strings :-D

[rugk]

I did not do so… :cry:

[rugk]

Maybe a friendly bot, considering that they are easy to translate… :wink:

Could I somehow filter all those belonging to "orchard" to review them? (Maybe using tags, you currently only have a "html" one…)

Also it's sad I cannot see any stats, where (these strings) still need to be translated…

[westnordost]

Just go to the last page. If you hover over the values, it should also tell you who did the translation and when.

[rugk]

Could I somehow filter all those belonging to "orchard" to review them?

Ah, just using the "sort: updated DESC" now.

And no I cannot see who did it. Maybe only the admin can see this.

[rugk]

Okay, review finished. And it was good to review them, some mistakes had to be corrected. And I am a crops pro now… :stuck_out_tongue_winking_eye: