Improving the Processing Algorithm Documentation and Python API

[romariocarvalhoneto]

QGIS Enhancement: Improving the Processing Algorithm Documentation and Python API

Date 2020/05/21

Author Romário Moraes Carvalho Neto (@romariocarvalhoneto)

Contact romariocarvalho@hotmail.com

maintainer @romariocarvalhoneto

Version QGIS 3.16

Summary

The Cookbook is a wonderful way to help new QGIS developers to get started, but the QGIS Python API is complex and may seem confused for someone that is just starting. Would be good to have for the Processing Algorithms Tools (which a beginner developer would be used to) a documentation that explains its syntax and that shows a sample of code for its use. This proposal will enrich the Processing Algorithms Documentation, the QGIS Python API and the Cookbook Cheat Sheet by adding links between them as well as missing information regarding to:

1) a Summary (e.g. found at Processing Algorithms Documentation on QGIS User Guide); 2) a Discussion/Usage (e.g. Processing Algorithms Documentation and Cook Book); 3) Syntax (e.g. QGIS Python API); 4) code example (e.g. Cheat Sheet and other parts of official documentation, other source of samples);

The aim is to have it for every processing algorithm tool, a path that may connect through all three QGIS official documentation (when necessary) to provide a smooth way to conduct the beginner from the cookbook to the QGIS Python API. This proposal may embrace three of the eight examples of topics: Updating and improving documentation; Improving API documentation; Helping new QGIS devs to get started with improved developer documentation and utilities.

The Need For Syntax Along With Code Sample For Beginners

When I started to program for GIS uses (about 7 years ago) I got used to the way the documentation of ArcPy was (and still is), explaining what a particular function would do, explaining its syntax, arguments types and with a clear simple code sample of its use. With basic Python and knowing what I need to do, I could look at the documentation and do it. When I started to work with QGIS, I got lost with the difference between the ArcPy and the QGIS documentation. Something what took me few minutes to perform with ArcPy, now took me hours of research and attempts. Things may get even harder when English is not your first language. So if we have in the official documentation more samples of code along with its syntax (API), especially for every processing algorithm tool that the user is already used to, the experience for those that are looking to start or already have an intermediate knowledge of Python will be greater.

Proposed Solution

Add sample of code and links between documentation (QGIS Python/API, QGIS User Guide and PyQGIS Cookbook) providing a path to have: Summary, Discussion/Usage, Syntax (link to the QGIS Python API) and code example for every for every processing algorithms. The steps will be:

1 - Copy examples from https://webgeodatavore.github.io/pyqgis-samples/ into the official API documentation, by placing these examples into the class doxygen documentation. This also benefits the c++ documentation, and keeps the examples close to the code so that they can be easily updated and revised if the API changes. 2 - Improve the Processing algorithm documentation by adding Python code examples showing use of the algorithms 3 - Add links between documentation.

Example(s)

Here we have a brief comparison between the QGIS processing algorithm documentation and the ArcGIS Toolbox documentation: Comparing the documentation the ArcGIS docs at https://desktop.arcgis.com/en/arcmap/10.3/tools/analysis-toolbox/clip.htm, an the equivalent in QGIS would be the Processing algorithm documentation (not the Python cookbook or the API docs). I.e. https://docs.qgis.org/3.10/en/docs/user_manual/processing_algs/gdal/rasterextraction.html?highlight=clip We can notice a the difference in the presentation of the information to the reader (e.g. the QGIS ones don't include any Python code examples on how to run those algorithms). The ArcPy documentation tends to have a pattern of Summary, Discussion/Usage, Syntax and code example. This pattern seems to make things clear to a beginner developer, specially the syntax and the samples of code.

Affected Files

(required if applicable)

Performance Implications

(required if known at design time)

Further Considerations/Improvements

(optional)

Backwards Compatibility

(required)

Issue Tracking ID(s)

(optional)

Votes

(required)

[nyalldawson]

To be honest -- looking at the cheat sheet there's a lot of really niche and unnecessary snippets in there. If the page is intended for use as a beginners guide to PyQGIS, then these really should be moved elsewhere.

Examples include:

• Change Look & Feel
• Change icon and title
• Map Update interval
• Refresh layer at interval

There's also issues with the existing snippets, e.g.

• Adding new feature without feature form: this should be using an edit session on the vector layer, not adding direct to the data provider
• Move geometry: refers to a "poly" variable, but geometry is a point!
• Refers to "composer" instead of print layouts

Before any work is done here, a review of the existing snippets needs to be conducted first to determine whether or not they are generally useful, correctly demonstrate best practice, and whether they are correctly targeted to beginners.

Additionally - for snippets which ARE valuable, these should be copied from the documentation into the code documentation to as Python code examples. This would then allow them to be shown in the official PyQGIS API reference docs.

[romariocarvalhoneto]

Really great points @nyalldawson. I think the idea of this cheat sheet (as other cheat sheet around) should be for a quick source for consulting. And since we are talking about a Cookbook section, it should aim the beginner that "went that far". You really point out the necessity of a review of its content and a better way to demonstrate best practice. I totally agree. There are good things in it, but some seem pretty useless for me.

[rduivenvoorde]

Thanks for having a look at it, everybody willing to help and restructure is great. Doing it in small steps is indeed the best way. The cheatsheet I think has become more of a set of examples which were hard to place somewhere else. Maybe move the really 'beginners' stuff to a page 'Get Started' is better?

@romariocarvalhoneto just to be sure everybody knows: all code in the pycookbook (and cheatsheet) is recently reviewed, and tested to be working with current version of QGIS. It is tested DURING the automated build and Pull Requests.

But please be careful to remove stuff, as others put considerable energy in it to make it work... Also because this little piece of code that one person thinks is helpful can be very valuable for somebody who just needs this few lines or insight. Moving around off course is ok.

[elpaso]

I agree with @rduivenvoorde, some of the examples there are of great value. But I also agree that they could be better moved elsewhere: after my code revision (that was limited in scope to automated code tests) I suggested to move some examples from the cheatsheet to the relevant section in other chapters, especially the advanced legend tree manipulation. Just note that it won't be easy: the examples rely on each other and on an existing tree configuration, it took me a considerable time to make the code pass the tests.

[romariocarvalhoneto]

Thanks @rduivenvoorde. I really appreciate your advice. The proposal was never meant to delete anything. At the application form for the grant, there is a question if the work is "starting from scratch or are do you plan to build on the work of others". I wrote that its built on work of others, surely. Even my schedule was built on each of the already existing things in it, to change as little as possible of the original work and just upgrade it. Thank you again to have me clarify this. We can sure think (or not) to move things around.

[romariocarvalhoneto]

Ciao @elpaso. Thank you so much for your advice on how hard was for you to get it working. Things get more "precious" when we have the good work from people like you and others. And yes, there are great examples in it. Actually, they may all be important and may be useful for somebody. But at the perspective of a beginner, some may be a little off there. One exemplo is the "Decorator" part of it. It's a beautiful piece of code. When I saw it, I just had to try it! But after I tried, I realized that would not be as useful (for me at the time) as the other things in it and may not be useful for beginners as well. I think it could be better explained, with screenshot of it, to illustrate what is the code for. Things like that.

[romariocarvalhoneto]

It seems that all of us, so far, agree that some things should be moved around. My suggestion in the proposal was to add more documentation on it and in a specific way, the "ESRI/ArcPy way", illustrating it and giving more importance to the things I think a beginner would use most, eg. the tools in the toolbar (the first things a QGIS users get aquented to) in the processing algorithms part and the other things already in it, as layers, TOC and these things we know will be searched in a Cheat Sheet. What topics of the Cheat Sheet you guys think could (or should) be moved, and where?

[Saijin-Naib]

To lend support, I hear/see a LOT of people layering praise on ArcPY and its documentation whilst lamenting QGIS' analogues. This can be seen/heard around my Twitter, GIS.SE, and other locations. Even locally within my org. the two prior programming co-ops did everything within ArcPY, even when interfacing with data outside of ArcGIS, because of how "easy" it was to work with.

As a complete Python/programming idiot, I can say with certainty that the current programming docs make my eyes glaze over and then bleed. I can't follow what is written, and they feel like an insurmountable task.

Conversely, a few minutes with the ESRI docs and guide, and I have the shell of a basic tool to use within my org.

I can't elucidate exactly what differs, but as a complete beginner, I can assure you that they are very different.

[romariocarvalhoneto]

@Saijin-Naib . QGIS documentation is getting better lately and I share some of your feeling about the "hardness" of reading its API, for example. It being a "wall" for beginners. But I really feel that the QGIS community is aware of it, and because of that is focusing energy on this 2020 Grant for:

• Updating and improving documentation
• Improving API documentation
• Helping new QGIS devs to get started with improved developer documentation and utilities. Even if those point are just suggestions, they are very pertinent.

[nyalldawson]

@rduivenvoorde

But please be careful to remove stuff, as others put considerable energy in it to make it work... Also because this little piece of code that one person thinks is helpful can be very valuable for somebody who just needs this few lines or insight. Moving around off course is ok.

I've got to disagree here -- filling this page with tasks which are extremely rare doesn't add value, and actually makes things harder for new users.

I've been in the PyQGIS game for nearly a decade, and I've never ONCE had the need to change the map update interval via PyQGIS. That's not a task a beginner needs to know how to do. Even worse, it's a snippet which relies on settings manipulation which is NOT stable API. It's just teaching bad practice.

This is only example from that page, but honestly, the page does need to be pruned, and the extra content moved to a "advanced snippets" page.

[Saijin-Naib]

@romariocarvalhoneto , this may be unique to me and my skill level, but out of all the Python books and guides I've tried, I've had the most success learning from and applying lessons from Python In Easy Steps.

No joke, that's how beginner I am. I didn't need the first lessons about setting up and installing Python, but pretty much from that point on I needed my hand held, and it did an incredible job. It is clear, concise, and provides excellent examples via text, code snippets, and GUI snapshots so you know what you're looking at.

Obviously, not to be copied, but I believe the layout and general flow of this book could be a great model for actually beginner-friendly documentation.

A paragraph with an objective, tips/tricks to help with the item being discussed, and an end summary of what you did/learned.

I think the repetition of concept in picture and words helps reinforce it, especially when the material is new to you.

[andreasneumann]

To lend support, I hear/see a LOT of people layering praise on ArcPY and its documentation whilst lamenting QGIS' analogues. This can be seen/heard around my Twitter, GIS.SE, and other locations. Even locally within my org. the two prior programming co-ops did everything within ArcPY, even when interfacing with data outside of ArcGIS, because of how "easy" it was to work with.

Your statements are a bit funny to me: you are comparing the results of a multi-billion company with several thousand employees and several decades of a head start with a community driven project that has the funds of not even one-tenth of a per cent of aforementioned company and is relying on 1-2 people working mostly unpaid on documentation - and you really expect them to be of equal depth and quality?

Do not forget that QGIS is a do-ocracy: one can complain a lot about how bad things are, or one can sit down and help solve the issues. And even if you don't have the skills to help on the tasks, you can ask your management to invest the cost of 1 or 2 ESRI licences per year into QGIS documentation to enable other people with the skills to do some paid work around QGIS documentation. QGIS documentation is seriously underfunded, if large organizations using QGIS, or planning to use QGIS would do a concerted and targeted effort and invest into QGIS documentation, it would be quickly in a much better shape.

Don't get me wrong: constructive feedback is very useful and always welcome to us. But whining and complaining about how bad things are in QGIS doesn't really help a lot.

I often hear people tell QGIS: you only need to do this and that, and it would convince our organization or management to give QGIS a try. It doesn't work this way. QGIS is YOUR project. If you want to use it, you are part of the community and you are just as responsible as anyone else in the project about its well-being. Everyone can contribute - with time, expertise or funds - rich or poor - beginner or expert.

So please stop complaining and start contributing! If you are serious about contributing to QGIS (and I have that impression from your recent statements around github discussions) then you could for example try to coordinate with other organizations to make the QGIS documentation, much much better compared to today. You only need a couple of organizations or companies and each one investing a fraction of what they normally invest into ESRI licensing and you would do a huge difference and come a long way with modest funds!

Don't forget: other people before you did the same: QGIS wouldn't be where it is today, if people didn't invest into it's well-being and progress.

[Saijin-Naib]

@andreasneumann thanks for the great rant, but I think you misconstrued my tone and intent.

[andreasneumann]

@andreasneumann thanks for the great rant, but I think you misconstrued my tone and intent.

maybe - or maybe not.

People are often demanding this and that from Open Source but don't realize that it is their own responsibility to make it happen. Things don't happen magically just be demanding it.

QGIS is a project, not a product. You can't keep filing feature requests and expect that someone would magically take care of it, if you don't start contributing yourself.

There is a really good read at http://nyalldawson.net/2016/08/how-to-effectively-get-things-changed-in-qgis/ and http://nyalldawson.net/2016/08/how-to-effectively-get-things-changed-in-qgis-a-follow-up/

Also - the constant comparison between ArcGIS and QGIS is tedious. QGIS is it's own project. It is not the goal to replicate ArcGIS. There are things that are better in ArcGIS - there is no doubt - but there are also things where ArcGIS isn't doing such a great job, and therefore it is useful not to focus too much on replicating another product, but to keep on improving QGIS as an independent project oriented towards the needs of their users. Most of QGIS users and developers don't have access to ArcGIS and therefore can't follow your ideas and references, like ArcGIS is doing this and that, and QGIS should do the same thing ...

@Saijin-Naib - I am not ranting at you specifically. I understand that you try to contribute useful things, but people often do not understand the nature of "product" vs. "project" and how Open Source works in general.

[rduivenvoorde]

@nyalldawson

I've been in the PyQGIS game for nearly a decade, and I've never ONCE had the need to change the map update interval via PyQGIS. That's not a task a beginner needs to know how to do. Even worse, it's a snippet which relies on settings manipulation which is NOT stable API. It's just teaching bad practice.

Ok, agreed on that one :-)

But I think we are talking the same thing here. It is just what you define as 'cheat sheet', to me the 'cheat sheet' is not the thing you show a beginner, but exactly the stuff that it has now: things you only do once every 6 plugins, and always forget but know that it is 'somwhere in the cheat sheet'..

That is why I think to split this up in

• a 'get started' page for python/pyqgis beginners: which is a ordered set of easy to do tasks tailored to beginners and
• a working trivial python snippets page, collecting every kind of working pyqgis that does not fit in one of the other chapters

What you name them is up to the person doing it :-)

@romariocarvalhoneto I would say: create an example page the way you imagine the best pyqgis doc avaialable, and show that example to the bigger world. If you need help building the docs, just ask! Do not take too big steps at a time :-) And thanks for trying to make stuff better !!

[andreasneumann]

@romariocarvalhoneto I would say: create an example page the way you imagine the best pyqgis doc avaialable, and show that example to the bigger world. If you need help building the docs, just ask! Do not take too big steps at a time :-) And thanks for trying to make stuff better !!

I totally agree with @rduivenvoorde - it is very good if you want to contribute. Try to do smaller steps, honor what people before you did, and don't think that just because you are not interested in some more obscure workflow that the stuff isn't useful to someone else. Having more examples in the cookbook doesn't hurt anyone, even if only a fraction of the users need it.

If you are not happy with the cookbook from the perspective of a "newcomer", then start something new. Cookbooks in the real world also have different levels and not every cookbook is targeting beginners.

You can also start something completely new if you want, if you are not happy with the current approach of the Python cookbook. Heck it could even be a separate cookbook targeting developers coming from an ESRI background ;-)

Both @Saijin-Naib and @romariocarvalhoneto : I totally agree with you that it would be very useful to improve the PyQGIS documentation, but I wanted to give you some perspective that there had been really limited resources put into documentation so far. Maybe in a ratio of 1/100 (very rough estimate) compared to what people invested into development work in the QGIS project. It is often relatively easy to convince someone to invest into the development of a new feature, but much, much harder to invest into documentation.

[romariocarvalhoneto]

@nyalldawson ... If you need help building the docs, just ask!

Thank you so much for the support. Sure make us want to contribute more!

[romariocarvalhoneto]

If you are not happy with the cookbook from the perspective of a "newcomer", then start something new. Cookbooks in the real world also have different levels and not every cookbook is targeting beginners.

You can also start something completely new if you want, if you are not happy with the current approach of the Python cookbook. Heck it could even be a separate cookbook targeting developers coming from an ESRI background ;-)

Thank you @andreasneumann. Great advice!! Really really great. But there is one thing that makes me think... wouldn't it be better if we (every one) join strength and efforts to build QGIS up for all than keep writing own stuff? I get the point when you say that we can do our own thing , meaning we are free to do it and still contributing, but I think the official documentation has to be the best as possible. The very best! I really think the work on Cookbook is awesome, but can be better. When new people come in, is the first thing they look at. And I want to point out again that Cookbook does a great job with it. But I think we are missing a bridge between the Cookbook, that "serves" you with "ready recipes" and the "wild" Python API and the lots of PyQGIS Classes and its "alias". I think Cheat Sheet could do the job and still being a quick reference for programming. Thank you again. Its great to have people passionate for what are doing!

[nyalldawson]

@rduivenvoorde

a 'get started' page for python/pyqgis beginners: which is a ordered set of easy to do tasks tailored to beginners and a working trivial python snippets page, collecting every kind of working pyqgis that does not fit in one of the other chapters

agreed. BUT - here's the critical point -- I do not think it is a worthy use of time / funds to expand on the documentation for the "trivial python snippets" page. There's so little value in spending time writing large descriptive docs for niche snippets, when the users who DO need these would already be experienced/technical enough to not require in-depth documentation for them.

Again, coming back to the infamous "map refresh rate" example: if someone does have a need to change this through a python script, then they most certainly know their way around Python/PyQGIS/Qt and don't need ANY "hand holding"!

[nyalldawson]

@andreasneumann

People are often demanding this and that from Open Source but don't realize that it is their own responsibility to make it happen. Things don't happen magically just be demanding it. QGIS is a project, not a product. You can't keep filing feature requests and expect that someone would magically take care of it, if you don't start contributing yourself.

It's off topic, but FYI @Saijin-Naib is one of the "good guys". His contributions aren't widely known, but he has been instrumental in the efforts to reverse engineer and create working GDAL drivers for various GDB formats (see e.g. #173), amongst a lot of other valuable work he's been doing behind-the-scenes with me!

[andreasneumann]

@Saijin-Naib and @romariocarvalhoneto - I apologize for my rants here. No personal accusations had been intended here - and your ideas and contributions are most welcome!

But the essence of my rant is still valid: don't expect QGIS to ever be an ArcGIS clone - neither in the software nor in the documentation. It won't be one. There is a Taiwanese company that basically cloned ArcGIS (https://www.supergeotek.com/index.php/products_supergis_desktop/) and in the OpenSource world gvSIG did the same thing with the old ArcView 3.x series. I wonder if that Taiwanese company will get some visits from ESRI lawyers some day? They blatantly copied the old ArcGIS GUI completely (probably including all of their shortcomings in the GUI and tools along the way). They have even stolen the same icons that ESRI used in the old ArcGIS software.

I think it is a terrible idea to just clone/replicate another software. Projects and products should innovate on their own and not copy someone else's ideas and workflows.

And of course, with the very limited resources that QGIS as a project currently has, you can't expect it to be on par everywhere with the commercial alternative that is 1000 times as big, when it comes to available resources.

Keep up bringing your ideas - just be careful not to call other people's work, who invested their time, "crap" or similar (I know you didn't do that ;-) )

[romariocarvalhoneto]

@andreasneumann now I got confused here... Maybe I didn't let things as clear as I thought. The proposed was to follow the ESRI "way" of writing doc, that is - For each topic divide it in: Summary, Discussion/Usage, Syntaxes (maybe a link to the QGIS Python API) and code example. I also suggested to add images of the results. It's a way of thinking, of organizing presentation to the reader. The code, the examples, everything else is TOTALLY different between ArcGIS and QGIS! If it was the same, I would not propose this. Clone is clone! Is the same. But I don't see the error and follow something or someone that do a good job in something and not apply it for the better. Standards are set and followed always, everywhere. I can just see good coming from following good standards. I must say something: Last Saturday a taught Python for beginners on QGIS. ALL of them use ArGIS instead. Since I enter the University (2001) I see ArcGIS being used in the Universities of Brazil. Please note how big Brazil is and its numbers. I graduated, did my masters, did my Ph.D, all here in Brazil and people were still using ArcGIS in Universities. Then I went to Italy for my double Ph.D. and guess what? They were also using ArcGIS. NOW IS THE MOST IMPORTANT: People learn in ArcGIS and after they graduate, they are up "in the wild world", by themselves, with the alternative of 1- buying a licence to use ArcGIS, and this is just way far from the reality of many, specially because Real (our money) is just getting down compared to Dollars; 2 - Use a "cracked" version of ArcGIS (and many do it); 3 - Use other alternative to ArcGIS. And it is here that some come to QGIS. In the last 10 years, I've seen that QGIS is getting more space in the academy. That's because of the work of many good people like you that give their time, efforts and gifts to it. Thank you again. Is a group effort. While some are trying to do their best to make QGIS great "from the inside out", there are people like me out there that are trying to make QGIS great from and to the outside, spreading its use and its capabilities. I am now finishing a specialization in Geomatics and people are less using ArcGIS and QGIS is getting more space. But many still learning in ArcGIS and coming FROM it INTO QGIS. This is the reality of many people down here, and it has been for years, and will probably be for a while if we keep thinking this way @andreasneumann . Just close our eye to how things really are, will not do any better. That's why I think a documentation as the one I proposed would do a great work for this many people (and I just mentioned the students, the academy. I did't even say about the public institutions).

And please, just reread what you wrote:

... just be careful not to call other people's work, who invested their time, "crap" or similar (I know you didn't do that ;-) )

Where did I say that yours or any other person's work is crap? Then you say "I know you didn't do that ;-)"????????? I really believe you didn't read any of the comments. I'm sorry...

[haubourg]

@andreasneumann @romariocarvalhoneto I really think you both agree, but written communication is really a tricky beast, I was bitten quite a few times.

Let's keep up the good work folks, you rock !

[saberraz]

I am opining here in personal capacity and have no link with QGIS grant proposal:

The unwritten rule, if you are applying for a QGIS grant is that first and foremost you should have been an active member of the community. There should be evidence of your commitment to the project as a developer, translator, document author, translator.

It is more understandable, when for example @andreasneumann and others who have been involved in the project and have dedicated $$$ and time to get the project where it is, read criticism and grand ideas which need funding from project. This has happened in other threads related to the grant proposals too.

So, in summary, if you have a great idea, maybe the starting point is: talking and engaging with other community members, help the current documentation/efforts,...Once your good works become evidence, your criticisms and your ideas (for funding or otherwise) will be seen in a completely different light.

[andreasneumann]

Ok - this now really getting a bit out of control. Sorry about that. Let's stop here. I apologize again for my miscommunication. Let's focus on making the documentation better ;-)

BTW: the statement about "crap" resulted from this comment from @Saijin-Naib:

"To lend support, I hear/see a LOT of people layering praise on ArcPY and its documentation whilst lamenting QGIS' analogues. This can be seen/heard around my Twitter, GIS.SE, and other locations. Even locally within my org. the two prior programming co-ops did everything within ArcPY, even when interfacing with data outside of ArcGIS, because of how "easy" it was to work with."

That basically means that the ArcGIS documentation is great and the QGIS documentation is crap - in a bit nicer wording - doesn't it?

And I was trying to explain that I agree that the documentation should be improved and the reason for the QGIS documentation being not as good as the ESRI one is that we have several factors fewer resources available.

Then I side-tracked a bit too much in my ranting - because people are often complaining that QGIS doesn't work the way that ArcGIS works. I was trying to explain that it is absolutely not the intention of QGIS to replicate what ArcGIS does.

But I also didn't find the correct wording ... and such issues are better discussed outside of a QEP.

As @haubourg mentioned - we are on the same boat here. We all think that the QGIS documentation should get some love - and this what this QEP is about.

[andreasneumann]

@romariocarvalhoneto - to help the case here: could you please add an example entry how you would envisage the new documentation structure and style to look like? I think the section "Proposed Solution" is a bit too brief as it stands now.

Most voting members here that will prioritize the proposals have no ESRI experience and it would help their judgements if they would see an example entry.

[andreasneumann]

@romariocarvalhoneto - to help the case here: could you please add an example entry how you would envisage the new documentation structure and style to look like? I think the section "Proposed Solution" is a bit too brief as it stands now.

Maybe - if the ESRI API documentation is publically available, then you could add an example link to an entry where you think it might serve as a good example. And/or a screenshot or a template.

[romariocarvalhoneto]

@andreasneumann I just got little confused because of my name in it:

@Saijin-Naib and @romariocarvalhoneto - I apologize for ...

I really really thank your work, everybody's work, and I'm trying to enfatize that in all comments. Me, personally, am a better person today because of the work you and others have done and am trying to do my part, as I can. Thank you again for the advises.

Maybe - if the ESRI API documentation is publically available, then you could add an example link to an entry where you think it might serve as a good example. And/or a screenshot or a template.

I will look up and example and try to show how it would be for a QGIS user. Again: its just a "try" for helping anyone coming from ESRI to stay definitely in QGIS, by seeing that they can do anything here. That they can bring all the research they've done and apply also here. Everybody wins.

[romariocarvalhoneto]

EXAMPLE FOR THE PROPOSED SOLUTION

I will try to explain the problem with a very simple example. The "clip tool". Lets pretend I need to clip a raster or a vector with PyQGIS. I go to google and type: arcpy clip. These are my three first search results: https://desktop.arcgis.com/en/arcmap/10.3/tools/analysis-toolbox/clip.htm https://pro.arcgis.com/en/pro-app/tool-reference/data-management/clip.htm https://pro.arcgis.com/en/pro-app/tool-reference/analysis/clip.htm Just look at them and see how they describe the "clip". They follow a summary, an illustration, usage, syntax (with an example how to write it: Clip_analysis (in_features, clip_features, out_feature_class, {cluster_tolerance})), and code samples.

Now lets type on google: pyqgis clip. In this example I give, we have a lot of results coming from other's people examples and code. Great, but I'm looking for something "official". Then I look for the first official result: https://docs.qgis.org/2.8/en/docs/user_manual/processing_algs/qgis/vector_overlay_tools/clip.html Is an out to date link (that, by the way, have the same ESRI "way"). No problem, we just click on the "Find the latest one here" and we go to the https://docs.qgis.org/3.10/en/docs/
We then search on the page for typing "clip". To me, the first examples are not exactly "PyQGIS". I found on the fourth: https://docs.qgis.org/3.10/en/docs/user_manual/processing_algs/gdal/rasterextraction.html?highlight=clip seems that it could be what I'm looking for, but I just don't undertand it. Ok. Someone may say "we are not at the Python API for QGIS". Lets look there. https://qgis.org/pyqgis/master/ I searched for "clip" again and the first result is: https://qgis.org/pyqgis/3.10/core/QgsClipper.html?highlight=clip#module-QgsClipper

I think now we can compare this link, to this, for example: https://desktop.arcgis.com/en/arcmap/10.3/tools/analysis-toolbox/clip.htm

I wish the QgsClipper could be more like the ESRI way of presenting doc to reader. My proposal is to do it for all the items in the Cheat Sheet. If not for all, at least for the most used ones, thinking on the people that want to start to code in Python with PyQGIS and I am sure that this way will be even more clear for anyone, not only for those that use Arcpy. I did this example of searching because this is what many people do. I am not questioning the efficiency of the searching tool or anything like that, but I'm try to show how good would be to have a official place with the most used tools and modules with examples that could help to pass from the Cookbook to the PyQGIS API.

[andreasneumann]

So we have two "official" documents for PyQGIS:

• the API documentation at https://qgis.org/pyqgis/master/
• the cookbook at https://docs.qgis.org/testing/en/docs/pyqgis_developer_cookbook/

Everything else is just stuff on the web, which might be useful or not. It might also be outdated or not.

So I think for your proposal it would be useful to involve the people who did the work on both of the aforementioned documentations. @3nids worked recently on the Python API documentation and @elpaso worked recently on the cookbook.

I am still not quite sure whether your ideas would better fit into the API documentation or the cookbook. Or maybe the two could be better connected with each other?

The API documentation is auto-generated, but maybe it could be extended with additional manual content (examples, illustrations, explanations, etc.). The cookbook is manual work, but according to @elpaso the examples are now all automatically tested - right? And there are already cross references from the cookbook to the API documentation. But as far as I know, the API documentation doesn't make references to the cookbook. @romariocarvalhoneto Is it this, what you miss?

@elpaso and @3nids - since you both worked on the API doc and cookbook. What are your thoughts? To me ,"the ESRI way" if I understand it correctly, is a combination of both the API documentation and the cookbook in ones single resource.

[DelazJ]

Hi Happy to see more people willing to improve the docs. More than welcome @romariocarvalhoneto. Thanks I'd like to clarify some elements in your latest comment. I was reading the discussion for days and was quite focused on the cookbook (and it would deserve restructuring for sure) or the cheatsheet, and I fail to find the reference to them in the example you provide. It looks like it's more about Processing algorithms. IMHO We need to clarify what we are talking about.

Then I look for the first official result: https://docs.qgis.org/2.8/en/docs/user_manual/processing_algs/qgis/vector_overlay_tools/clip.html Is an out to date link (that, by the way, have the same ESRI "way").

By Esri way, you mean we have the rights sections for the information (description, parameters, console usage...)? Because there is nothing to read in that example. :smile:

To me, the first examples are not exactly "PyQGIS". I found on the fourth: https://docs.qgis.org/3.10/en/docs/user_manual/processing_algs/gdal/rasterextraction.html?highlight=clip seems that it could be what I'm looking for, but I just don't undertand it.

Normal that you do not understand, this is the clip raster by extent tool, not the right corresponding tool for the 2.8 url you provided. vector clip was few lines below, at https://docs.qgis.org/3.10/en/docs/user_manual/processing_algs/qgis/vectoroverlay.html#clip And I think we have done great jobs these last years: description and parameters, inputs, outputs are filled; we have a screenshot that explains what it does. I'm rather proud of what this page looks like. Wish there are more like this. What is missing, compared to 2.8 is that there's no more console usage. Why? Because QGIS evolves so quickly that the codes were outdated (they were from 2.x) and with the manpower we had it was better to drop them than trying to update with the risk of unnoticeable breakage. Providing individual examples was not possible so we provided kind of "how-to" information, eg in https://docs.qgis.org/3.10/en/docs/user_manual/processing/console.html Today, we have setup automatic checks for code sample so a console usage section makes sense because it can be maintained "easily"

@andreasneumann

To me ,"the ESRI way" if I understand it correctly, is a combination of both the API documentation and the cookbook in ones single resource.

I may be wrong but from the provided examples, this is more about extending the processing documentation with code samples and context than linking pyqgis and cookbook.

[romariocarvalhoneto]

By Esri way, you mean we have the rights sections for the information (description, parameters, console usage...)? Because there is nothing to read in that example.

Hahaha. Yes that is what I'm talking about. The sections. It's like a "standard", a "pattern".

And I think we have done great jobs these last years: description and parameters, inputs, outputs are filled; we have a screenshot that explains what it does. I'm rather proud of what this page looks like. Wish there are more like this.

You are being modest. The work have been more than that! It's a being a wonderful job. Congratulation! And I follow the documentation pretty close, I would say, and I see changes being done almost everyday lately. The overall work done in the documentation is going in the right way, definitely.

@andreasneumann

To me ,"the ESRI way" if I understand it correctly, is a combination of both the API documentation and the cookbook in ones single resource.

I may be wrong but from the provided examples, this is more about extending the processing documentation with code samples and context than linking pyqgis and cookbook.

The initial idea was to provide richer documentation on all the topics in the Cheat Sheet, focusing more on the processing tools and the most used examples that are already there.

@DelazJ , I proposed that because I think is a small step (I hope I'm not wrong). From my background, I think a richer documentation focused on the things that new users use the most (as the tools, and other topics on the Cheat Sheet) in a way that can be consulted quickly, will make wonders for the new developers.

[romariocarvalhoneto]

@nyalldawson , @andreasneumann , @elpaso , @3nids , @DelazJ and others that are participating. Thinking about all the discussion we are having, I came into another idea, and I hope is not off the trail, but may be an alternative, if working on the Cheat Sheet seems not the write thing to do. What if we create a new session on the Cookbook, I would say, focusing on "How to read the QGIS Python API" then we could choose some examples from the tools (processing) or from the Cheat Sheet, or any other we choose. Would it be better or the original idea still the best one? I don't think we have a place on the official documentation that talks about that. Have we?

[nyalldawson]

Taking a step back, I think we're mixing up a few concepts here:

1. The Python/API documentation
2. The Processing algorithm/ArcGIS toolbox documentation
3. The PyQGIS cookbook

1. API Documentation

QGIS: https://qgis.org/pyqgis/master/ ArcGIS: https://desktop.arcgis.com/en/arcobjects/latest/net/webframe.htm#welcome.htm

Specific examples: QgsMarkerSymbolLayer: https://qgis.org/pyqgis/master/core/QgsMarkerSymbolLayer.html#module-QgsMarkerSymbolLayer ArcObjects MarkerSymbol : https://desktop.arcgis.com/en/arcobjects/latest/net/webframe.htm#IMarkerSymbol.htm

We're basically on an equal footing here, possibly QGIS has a slight edge (most of the ArcObjects documentation is of the form "setSize: sets the size", whereas with QGIS you do get a bit of extra detail sometimes).

2. Processing algorithm documentation/ArcGIS Toolbox documentation

Now, when you compare to the (excellent) documentation at https://desktop.arcgis.com/en/arcmap/10.3/tools/analysis-toolbox/clip.htm, the equivalent in QGIS would be the Processing algorithm documentation (not the Python cookbook or the API docs). I.e. https://docs.qgis.org/3.10/en/docs/user_manual/processing_algs/gdal/rasterextraction.html?highlight=clip

Comparing the ArcGIS docs vs the QGIS Processing algorithm documentation, yes, there is clearly a large difference in quality. (E.g. the QGIS ones don't include any Python code examples on how to run those algorithms)

3. PyQGIS Cookbook

This is neither a reference guide nor a user manual -- it's closer to a text book/"dummies guide" to PyQGIS.

Honestly, targeting the PyQGIS Cookbook would be the lowest priority in my view. It's much more important that we improve the official API reference guide and the Processing algorithm documentation, so I would personally prefer that this proposal targeted one of these two.

In my (humble) opinion, the priorities would look something like this:

1. Copy the excellent examples from https://webgeodatavore.github.io/pyqgis-samples/ into the official API documentation, by placing these examples into the class doxygen documentation. This also benefits the c++ documentation, and keeps the examples close to the code so that they can be easily updated and revised if the API changes.

2. Closing off tickets from https://github.com/qgis/pyqgis/issues , so that the PyQGIS documentation site is easier to read and cleaner

3. Improve the Processing algorithm documentation by adding Python code examples showing use of the algorithms

4. Extend the PyQGIS Cookbook

[alexbruy]

Improve the Processing algorithm documentation by adding Python code examples showing use of the algorithms

Nor sure about this one. Some time ago we had Python oneliner for each algorithm showing how to call it from console/code. But these snippets quickly became outdated (if algorithms were updated) or missed (when new tools were added).

Also this infomation can be easily retrieved by user. Looking at the Processing history is enough.

[elpaso]

I cannot say anything about the QGIS/ESRI comparison because I've never used an ESRI product in my life, but having seen the links to the example ESRI docs I must say they look very good!

@andreasneumann your summary of the current API docs/cookbook situation is correct, I (we) have though about merging API docs and cookbook several times in the past (you can find a few untested code samples scattered in the C++ class headers now) but this matter may need another separate discussion. My quick take on this topic is that even if I would really like to see a unique source of truth for API documentation and the cookbook (and I see the advantages @nyalldawson is pointing out), C++ class headers are IMO not really suitable for examples. The issues I see with that approach are that:

• it would raise the barrier for API documentation contributors that are not proficient in C++ QGIS development (doxygen has its own quirks and the C++ PR approval is hard, even when you change a comma in a comment)
• embedding images (if we are going to) would be hard
• testing snippets would be difficult (this is the real blocker IMO, but maybe I just ignore the way to do it)
• organizing material in a hierarchy of chapters and sections would be very difficult In other words: turning doxygen C++ class headers code snippets into a book-like structure like the cookbook seems to me too difficult to achieve and to maintain, even for a C++ developer.

So, coming back to the original proposal here, I like it. But I have some questions, concerns:

1. the cheatsheet is currently a chapter of the cookbook where small snippets of code were thrown without much organization and description, more in a wiki-style than in a book style, while it is totally possible to turn it into an ESRI-style (pass me the term) one-page-one-function collection, I don't think it worth the effort because some snippets really would benefit from that but other would probably not. Such an effort would need to be addressed and designed from the ground-up by selecting the most frequently used API commands, like it is already done in the individual sections of the cookbook, for example: https://docs.qgis.org/testing/en/docs/pyqgis_developer_cookbook/vector.html#iterating-over-vector-layer, I doubt it that we have the workforce to make it happen.

2. turning a few code snippets into individual pages wold not be a game changer unless we find a way to make it grow to include a substantial amount of the most used-needed API functions (the most part of which is already documented in other chapters of the cookbook). So, what would probably be an ideal outcome is if we had a (sustainable => lot of people who is willing to contribute in the future) path forward to completion.

[nyalldawson]

I (we) have though about merging API docs and cookbook several times in the past

I'm -1 to this - I think there's a very good use case for keeping the cookbook as a structured guide to starting with PyQGIS. The API docs will never be this.

[nyalldawson]

the cheatsheet is currently a chapter of the cookbook where small snippets of code were thrown without much organization and description, more in a wiki-style than in a book style, while it is totally possible to turn it into an ESRI-style (pass me the term) one-page-one-function collection

I'm also -1 to this - I think the maintenance burden would be just far too great.

[romariocarvalhoneto]

Improve the Processing algorithm documentation by adding Python code examples showing use of the algorithms

Nor sure about this one. Some time ago we had Python oneliner for each algorithm showing how to call it from console/code. But these snippets quickly became outdated (if algorithms were updated) or missed (when new tools were added).

Also this infomation can be easily retrieved by user. Looking at the Processing history is enough.

But isn't there a way to try to "bound" together the documentation and the whole process of "algorithms being updated or new tools being added"? For the future, I mean. I think this should be one thing. Would make things just so much easier for every one. When something is changed, documentation must be changed as well.

[romariocarvalhoneto]

Taking a step back, I think we're mixing up a few concepts here: ...

@nyalldawson , just awesome what you wrote!

2. Processing algorithm documentation/ArcGIS Toolbox documentation

This is exactly what I meant to do, based on someone (actually many) that is coming from a normal QGIS user "stage" and wants to start to code (this is an example). The experience he will probably have at that point, would be the processing algorithm tools and somethings with layers and TOC, I would say. Things that are visually given by the "QGIS buttuns". These things are in the Cheat Sheet, and that's why I proposed to change there. But we may do it in any other way, anywhere else. I think things are very clear now. Thanks @nyalldawson and all that helped clarify things.

[nyalldawson]

No problem! Will you update the proposal as a result?

There's still the (valid) concerns raised by @alexbruy about including Python snippets in the Processing documentation.

[DelazJ]

There's still the (valid) concerns raised by @alexbruy about including Python snippets in the Processing documentation.

Technically speaking, wouldn't the doctest infrastructure cover this? That's not different from having code sample in the cookbook chapter, is it?

[andreasneumann]

No problem! Will you update the proposal as a result?

Yes, I think this proposal needs to be reworked. Ideally, the bulk of the text and information should be in the sections "Proposed Solution" and "Examples". Currently, the "Proposed Solution" section is way too short, and the other section "The Need For a Clean and Direct Documentation for Beginners" might be shortened a bit (not strictly necessary, but it would be better balanced then).

It would be great, if the discussion here could be summarized in this proposal, and the available three main documentations (API documentation, PyQGIS Cookbook, Processing algorithm documentation) mentioned to provide the wider picture where this proposal fits in to the QGIS documentation landscape, but then narrow down to the area where you want to work within the current various documentation projects.

There is still time until Friday to improve this proposal, as the deadline was extended until then. It really needs improvements, otherwise I am pretty sure voting members would ignore this proposal as it doesn't provide a clear enough plan.

[romariocarvalhoneto]

No problem! Will you update the proposal as a result?

Ok. Updated. Even if I still want to update the information on the Cheat Sheet, seems like now the focus is more towards the processing algorithms documentation and the Python API. Should I change the Title? I mean, can I? Do I have to resend the application form, or just change the name (if needed) since at the first form there is a link to this QEP?

[andreasneumann]

Yes, I think it might be useful to change the title. Maybe something along "Improving the Python documentation part xy (whatever you really target now)" - and maybe remove the "as ESRI/ArcPy" part. It is good enough to mention that part in the proposal.

@anitagraser - what do you think, should this be resubmitted because the title will change and also the proposal changed a bit during discussion?

[anitagraser]

I don't think it's necessary to resubmit. The discussion phase is meant to cause readjustments and we have the QEP link in the submission.

[3nids]

proposal has not been marked as eligible for grants 2020 see https://lists.osgeo.org/pipermail/qgis-developer/2020-June/061439.html

[jinglelocks]

I would like to see some movement on this. The documentation is absolutely dire for the beginner. Once I've consolidated some of the basics I've learned I will gladly contribute, but really how hard is it for someone in the know to knock up a page or two of essential knowledge for running a processing algorithm?

[Saijin-Naib]

I've not much experience, but I do write documentation for another FOSS project and I can say that at the least, I find it incredibly difficult.

What I imagine might make it even more difficult is that as one gets more experience and knowledge with the program and gets further away from their beginner state, they might start to forget or lose clarity on what things tripped them up or need to be explained in more detail.

[esnyder-rve]

Was this work actually completed? I'm currently working on a proof-of-concept new design and layout for processing algorithms in the user manual.