Considerations about Geonode installation process

[danieldids]

Same history again today: sudo apt-get install geonode on a clean and upgraded ubuntu 16.04 server VM does not work. :-(

I'm trying to deploy geonode at my job (a public Brazilian institution) to serve as a SDI since version 2.4. But every version suffers from the same kind of problem, which is always related to the installation process or bug fixing. So we are always deferring its use, since it's going to be used by lots of other institutions as well.

With other system deployments I've done that depend on databases or other systems, those pieces must be installed and configured separately - one example being ownCloud: install DBMS, create user and database with appropriate permissions, install apache, finally install ownCloud and provide user name, password, DB IP:port, etc. for its wizard install process.

Geonode downloads and installs everything and tries to make it all work. Any problem along the way, you're just screwed. Any problem with the upgrade process, you're screwed and sometimes you loose your data. The detailed installation instructions on geonode.org are sometimes not updated, or not complete. The installation process asks for a clean machine (linux in my case), which by itself is a very scary demand for a sysadmin as sometimes I have to be at my work. local_settings.py has plenty of configurations that can cause the system to break, and sometimes I'm not sure of where things should be set.

So, my suggestion about geonode install is: keep it simple. Make requirements as install postgres and postgis version x, create DB and user, install tomcat[x], download, install and configure geoserver version x for your needs, finally run apt-get install geonode and provide the required credentials.

The manual installation process is pretty much what I've just said - one would suggest. No, it's not. It requires lots of manual settings, coping stuff around, and still it doesn't always work.

I don't know if django requires all this mess, but I feel it does. It's not good this way. Such complex and big systems (as geonode already is) need to be more encapsulated, they should work by exchange of simple messages around (REST for instance, which is already available for geoserver). They should be pluggable by a "few cables" only, so any issue can be identified more easily.

Anyway, those are my opinions. I really would like to use geonode, but it still seems to be not that reliable yet (for my needs).

[capooti]

I have a huge respect for the people which have been and are working with GeoNode package. But I can't agree more with you. Furthermore I think almost all of the production GeoNode instances are not running from the GeoNode package, but using the development installation.

What is really nice of having the package is that it lets anyone to have a quick install and test it. But I think at this point a better option is OSGeo live - which I have recently used it and works well for demo/learning purpose.

just my 2 cents

[giohappy]

You're right, the "easy way" of installing a GeoNode instance is not that easy. And I also agree that in the future the architecture should adopt loosly coupled patterns (modular services with APIs), and this was also discussed during the recent summit in Turin.

However this is a community driven open source project. Every present and past contributor is doing its best to develop, mantain it and give support. Every contribution is well accepted, be it coding, docs improvements or funding. Your institution could for example contribute with a specific interest in improving the packaging and the setup / deplyment workflows.

In our experience we never setup GeoNode through apt packages. We usually set it up piece by piece, or using Docker (more frequently now), or with paver for local testing setups. Too many ways maybe, sone poorly docunentated but, as I said, come and join the effort ;)

[danieldids]

@capooti , for quick tests, I think a demo site is always a better solution. Geonode and many other projects already have a demo site, so this point about quick testing is resolved.

Why can't geonode install be more like dragging files and dirs into /var/www/geonode/ and setting up database user name and password? Is it because of django?

[danieldids]

@giohappy , it may seem that I'm complaining about an open source and free software project. I am. :-D

My contribution up to the present was posting issues about my testings (which were many), and trying very hard to make geonode work by package or manual install.

I'm complaining because after all this time, after all my efforts to make it run, after so much data being fed into geonode 2.6, and now with an upgrade process which left me with a broken geonode install (VM snapshot to the rescue), I simply cannot understand why running geonode should be so difficult! I like the project very much, but again I cannot rely on it.

I know you people are not responsible for my failure in any way, but today (when the upgrade broke my geonode install) I felt I had to tell my point of view about this whole thing.

[giohappy]

I may understand your frustation, and your feedbacks are welcome, but I will repeat myself: a proactive contribute would be much more helpful ;)

PS: it's not a PHP script that you can throw in a /var/www folder of an (PHP enabled) HTTP server. It's way more complex, and it's not a preference, it's how many applications work, not only those using Django...

[danieldids]

(this whole conversation appears to be becoming a discussion, which was not my intention, I'm sorry)

I'd like very much to help developing geonode, but for that I need first to satisfy my boss with a working SDI based on Geonode. :-D

My comment about geonode running somehow like dropping files into /var/www is because some other systems I've deployed work very much like that, and they are not just PHP (blerg!) scripts, they are also complex. Since I know little about django (again, I didn't have time to learn it), I just don't know how it works, whether this approach would be possible using it.

Again, my point is: I really think geonode should be more loosly coupled - as you also stated and I'm happy to know that you guys are also concerned about this.

BTW, version 2.8 is amazing!

[giohappy]

We (and others here) can offer professional support for you and your institution. Maybe it would help. Meanwhile a couple of suggestions:

• try the setup with the pavements commands to start experimenting.
• report errors, logs and whatever can help us to understand what's wrong with your setup
• for upgrade and data migration have a look to the related docs, in case you missed them

I didn't mean to start a discussion. I just wanted to turn your frustation into sonething helpful ;)

[lucernae]

I can relate to your frustations (I'm not a core Geonode developer, btw). I also had some several production instance running and you had to tune up your data carefully between upgrades (and of course saves some backup). Despite all that, we are trying to contribute back, because this project had many potentials to improve. In my company itself (Kartoza), we install and setup production instance using Docker + Rancher. We even deploy from development code and fix bug as we go along. This is of course not ideal for user who just want to install GeoNode and then be done with it.

But in our end goals, after we finished experimenting with Docker + Rancher, we will try to pitch the same or similar installation process as alternatives for GeoNode community.

GeoNode is already a loosely coupled system in my opinion or at least heading in that direction. I've seen efforts of decoupling communication between GeoNode and Geoserver. Django also works sort of the same way with PHP, but to make it more robust in prod env we run it on top of wsgi with extra setup.

What we might want to improve is the TDD and BDD approach, because it had so many components we need to test every parts extensively, either for migration or clean install. But, I see GeoNode team also heading there.

I don't want to turn this into my extra rant :). Basically, we will try to reach that goal as a community. After all, I'm also using this project and learn a lot from it. If you have some spare time, you are always welcome to try again and provides us with more detailed error. This kind of feedback also helps us as a community.

[francbartoli]

@danieldids Perhaps your boss was looking for a cheap and working SDI and I can understand you were trying to do your best with GeoNode but I cannot figure out how you could expect a drag and drop deployment for such a complex distributed application. Again the strength is also the flexibility to allow the administrators to configure a lot of functionalities at their needs and this obviously has to be balanced with the complexity and a good understanding of what you want to achieve. I cannot believe you could end up to deploy any sort of software without the required knowledge to be familiar with any piece of that. That said just to leave the discussion and move this forward to a meaningful improvement proposal I'd like to know what you do mean with "keep it simple".

• Do you mean the documentation is too extensive and maybe making people confused how to approach with different components? I'm asking because we are aware by far it needs to be improved. Can you provide changes which improve it as pull requests?
• Do you have specific contributions on how to improve the current installation script https://github.com/GeoNode/geonode/blob/master/package/install.sh or maybe you can provide a better approach to follow?
• Do you have specific errors with the migration process that you can file and discuss with developers to improve and mitigate bugs within that procedure?

The aim of the Docker deployment is to make it simpler but that doesn't mean you can have it at no cost at all. You should learn Docker concepts, commands and then understand what is going to happen behind the scene.

Regardless your intention to learn django for sure we can improve together all the ways the deployment process is used to happen. As @giohappy already suggested there are a lot of ways this can be applied:

• Funding improvements (docs, scripts, packages) by asking to professional support
• Spending your time to provide pro-active feedbacks or contributions to documentation and code base even just for the deployment package. Or maybe adding/refreshing new/obsolete OS packaging (btw CentOS would be welcome)
• Also I would add the possibility to improve your knowledge of GeoNode asking your boss a training

Thanks, Francesco

[danieldids]

@francbartoli

Yes, I have some proposals for improving geonode installation process.

That said just to leave the discussion and move this forward to a meaningful improvement proposal I'd like to know what you do mean with "keep it simple".

Do you mean the documentation is too extensive and maybe making people confused how to approach with different components? I'm asking because we are aware by far it needs to be improved. Can you provide changes which improve it as pull requests?

The documentation is mostly good and comprehensive. I could improve it where MY installation process failed, but most times I was not able to point out where the problem exactly was. Sometimes, running apt-get twice would make geonode run, sometimes not. Why? I don't know, so I could not provide any update to the documentation. On top of that, I'm trying to run geonode behind a proxy, which makes things a little more difficult, specially because we don't have full control of it.

Do you have specific contributions on how to improve the current installation script https://github.com/GeoNode/geonode/blob/master/package/install.sh or maybe you can provide a better approach to follow?

By specific you mean actual scripts? If so, no, I don't. My suggestion about the approach is: require the sysadmin to install tomcat, postgres/gis, apache/nginx the regular way, create db/user for postgres, deploy geonode's geoserver.war file manually and them proceed with the installation process, this being the approach both for full manual and apt-get (or yum) installation process. Geonode should handle postgis configuration, not the user.

I say this because: 1) maybe the environment already has postgres and tomcat installed on dedicated machines, with other stuff running and properly configured for performance/storage/backup/etc; 2) once I've removed geonode (via apt-get) and geoserver/postgres were also removed (along with all the data). Fortunately, this was not a production server; 3) geonode.org already provides (abeit it is offline now) a demo site, so a quick installation is less of a concern; 4) by offloading pg/tomcat/apache installation instructions off your hands, you make your lives easier.

Do you have specific errors with the migration process that you can file and discuss with developers to improve and mitigate bugs within that procedure?

The migration breaking happened yesterday, and I was compelled to express my opinions here. Today I'm going to try making it run again, so I could file the problems I'm having.

I missed a chance to file a small fix when removing geonode 2.6 using apt-get remove - the apt-get script looks for a file in tomcat and when it doesn't find it, the process fails. I'll try to replicate that and I'll post it.

[Roburetto]

Forgive me for intruding but having spent the past few days trying to familiarize with Geonode installation I feel I have something to say.

As I see it the documentation seems to presume to be in the user’s best interest to install a development environment with all the trimmings and potential while most would probably settle for something that just works. I mean, it’s written from a developer’s point of view. Nothing wrong there but, from where I stand, neogeography and web mapping ought to be about empowering enthusiastic single individuals with the ability to produce, remix and share data and not necessarily institutions with dedicated staff.

A couple of things I find confusing in the installation process:

Why isn’t all the Python stuff handled through pip? What’s the point of doing it from a .zip you have to download using a bunch of procedures when you can just “pip install geonode”?

How come you have both a .war and a .deb for installing geonode-geoserver? Regardless, by following the documentation I would end up with duplicate data in tomcat/webapps and data/geoserver.

Anyway, laugh all you want of my naiveté, but I describe here in detail what ended up working for me, including the modified install.sh.

Thanks a lot for you work! It is much appreciated.

[afabiani]

Every contribution is useful to improve the project. Unfortunately active developers are currently very few, and it is almost impossible to test everything and integrate all improvements too.

However, a good start would be to clean up and refresh the documentation. As an instance we should remove outdated (or wrong/misleading) installation pages in favor of more simple processes like the one you described above.

[olivierdalang]

Related GNIP (Support production-grade deployement using Docker) : https://github.com/GeoNode/geonode/issues/3707

Would that approach make sense in your use cases ?

[danieldids]

I'm following the discussion. I'll just point out one comment that I liked:

@francbartoli If it results in making usage/maintenance significantly more complex, I'd favor leaving this out of the basic setup, since for large scale deployments, an advanced sysadmin will be involved anyways.

I think too those deployment options you are discussing are very welcome, but they add a layer of complexity which is not always needed. A big company will have more people available to manage stuff.