Conf console or first boot memory settings

[DocCyblade]

Would be nice to be able to set some tweaks for the server depending on what hardware it is installed on. Could be from the conf console and/or the first boot hooks

[l-arnold]

Agree. If one moves RAM up or down it doesn't seem to change the Swap File size. Could be an issue if you doubled your ram but not your Swap file in some situations.

[l-arnold]

Seems Complicated. Easiest right now would be to document the changes that can be done.

Would be lovely if ConfConsole did allow some changes of the sort however, I agree.

[JedMeister]

It would be cool if we had a simple headless bash script that would accept values (e.g. via switches) but if not passed values would check to see how much RAM was installed and use the formula for config using Odoo. We could run it as a firstboot script. Also perhaps it could email the user with what it has done and a link to where to find additional info!?

[DocCyblade]

@JedMeister - Mission accepted :+1:

So I found how to grab CPUs

 CPU_CORES=`grep -c ^processor /proc/cpuinfo`

and memory in bytes

PHY_MEM=`free -b |awk '/^Mem:/{print $2}'`

Reading https://www.turnkeylinux.org/docs/inithooks now :-)

[DocCyblade]

@JedMeister - Dang I just learned something I think I forgot or did not fully understand about the preseeding support.

also learned

. /etc/default/inithooks

is the same as 

source /etc/default/inithooks

learning too much too early in the AM

[DocCyblade]

@JedMeister - Not sure how to send an email, but the scripting I can do.

I was thinking, in an interactive script to generate new values for tuning and present this with a YES/NO/SKIP screen so they have the chance to OK the changes.

Trying to take apart the python file and see how the different dialogs work

[JedMeister]

TBH I think that so long as it runs reliably, a non-interactive script would be fine. The user can then manually tweak them later.

Having said that; allowing the user to accept or adjust them (and re-run the script to tweak them) would be quite cool. Possibly consider an interactive one for future builds?

Initially you could just send email to root@localhost. If the user has entered their email for the auto security alerts then it will be auto forwarded; if not then every time they log in it will alert the user with something like "You have mail"

[l-arnold]

I manually tweaked today. Not that complicated but I know how to do it.

I do think we should have the SAME GRID for "Had Limits" as we do for "Soft Limits" in the openerp-server.conf file.

Also, we must remember always that if the Admin Password is changed then Odoo will throw out all the Documentation that there is in we in there. As such we need to mirror the content in the Backup File whenever there are updates to the primary setup file.

[DocCyblade]

I do think we should have the SAME GRID for "Had Limits" as we do for "Soft Limits" in the openerp-server.conf file.

The issue is, I have not found good documentation on how this value works. I understand and can see the max memory for the soft but the hard limit, while I understand what it does, I am not fully understanding what and when theese setting would need to be change. I do know it seems to be based on virtual memory and not real memory. A trip through the source code would shed some light on it, plus reading up on memory management and use with the linux kernel would be added to my reading list.

Also, we must remember always that if the Admin Password is changed then Odoo will throw out all the Documentation that there is in we in there. As such we need to mirror the content in the Backup File whenever there are updates to the primary setup file.

Yeah, thats the bad part about that. I thought about forking Odoo repo and finding the code that changes that file and modifying it to only update the admin password, and leave the comments alone, then issue a pull request and put my mark on Odoo itself! I need to learn python, another reason to learn it

[l-arnold]

Well, I just found this (from launchpad documents for Odoo) I need to cut my "hard limit in half) as I was giving 1.5 gig to "all workers). It is a "per worker number" it seems.

  Multiprocessing options:
    --workers=WORKERS Specify the number of workers, 0 disable prefork mode.
    --limit-memory-soft=LIMIT_MEMORY_SOFT
                        Maximum allowed virtual memory per worker, when
                        reached the worker be reset after the current request
                        (default 671088640 aka 640MB).
    --limit-memory-hard=LIMIT_MEMORY_HARD
                        Maximum allowed virtual memory per worker, when
                        reached, any memory allocation will fail (default
                        805306368 aka 768MB).

https://answers.launchpad.net/openobject-server/+question/224433

[JedMeister]

What about and interim measure of a well documented /etc/openerp-server.conf.example?

Also my 0.02 :dollar: re the hard limit; I'm not sure that you would be able to document that much further as it will be quite dependant on the end users' server resources; users/server-load & performance desires.

[DocCyblade]

I think we have something like that, a template. Could rename it

[l-arnold]

This is my math:

If < than 1 GB Ram, then workers = 0 I -= > 1 GB, then Total Ram * 3/4 = Ram Avail for Odoo Min Soft Limit = 256 K Optimal Soft Limit = 640 K Min Soft Limit = (.375gb to (512 K) 512 may be (too high but Odoo says 640 mb default)?) Optimal Hard Limit = 756 K (.75 gb)

1 Gb ram = .75 gb avail = .75/2 = Hard limit( .375) gb hard and 256kb soft 2 workers (low for Odoo rec above, but sufficient for printing) 2 Gb ram = 1.5 gb avail = 1.5/2 = Hard Limit (.75 gb hard and 64 kb soft 2 Workers. (lines w/ Odoo defaults)
Additional Ram would increase workers but not increase Limits If certain "multi task jobs with finite resources were needed, then more workers could be tasked to smaller limit sets)

Pretty much what I think you have in everything Ken.

[DocCyblade]

The problem with the hard limit is it using virtual memory according to my testing. This virtual memory is not a real indicator of real memory being used nor swap space it includes all code and other libraries that it may need on disk that's why when we tried to print a PDF it will using 1.3 gig because it was using all of the other libraries needed

i'm going to do more research on this virtual memory and what you do uses that limit for and what is looking at the worker process so that we can better understand this limit and make a better educated guess

http://serverfault.com/questions/138427/top-what-does-virtual-memory-size-mean-linux-ubuntu

Sorry for the bad grammar I'm using my phone to translate.

[DocCyblade]

Another great read about virtual memory on linux regarding the top command

http://stackoverflow.com/questions/561245/virtual-memory-usage-from-java-under-linux-too-much-memory-used

I now understand what virtual memory is in top, now I need to look at the source code of Odoo and see how it's monitoring it's work or processes and what stats it's using to determine if the hard limit is reached this would give us a better understanding and way of setting that limit

[DocCyblade]

Started looking at the source code of Odoo found the below link that starts the threads and processes for workers. Still reading to the kids but I believe this is where the processes get created not sure if they get monitored here as I'm still reading

https://github.com/odoo/odoo/blob/8.0/openerp/service/server.py

[DocCyblade]

Ok so I found out ALOT. Read through the code (Python very easy to read) starting here https://github.com/odoo/odoo/blob/8.0/openerp/service/server.py#L698 looks like where the limits are defined, here https://github.com/odoo/odoo/blob/8.0/openerp/service/server.py#L715

That is where it is pulling in the hard limit. It's calling a function called setrlimit . I looked up the resource module https://docs.python.org/2/library/resource.html and from that I found that it's setting a limit specifically RLIMIT_AS This got me googling and found the linux command with the same name. This was very interesting and learned ALOT! This is what is killing the process if it goes beyond the hard limit.

If you notice, this limit is maxed out on a 32bit server since it's a long number. A 64bit server has no limit.

So as far as what is a mmap number that is "too large"? I started google some more and came accorded some article that I have not read yet but will tonight.

http://www.linuxjournal.com/article/10678 http://www.makelinux.net/ldd3/chp-15-sect-1 http://www.gnu.org/software/libc/manual/html_node/Limits-on-Resources.html http://www.javacodegeeks.com/2014/09/using-rlimit-and-why-you-should.html

https://docs.python.org/2/library/resource.html 
The maximum area (in bytes) of address space which may be taken by the process.

http://man7.org/linux/man-pages/man2/getrlimit.2.html

      RLIMIT_AS
              The maximum size of the process's virtual memory (address
              space) in bytes.  This limit affects calls to brk(2), mmap(2),
              and mremap(2), which fail with the error ENOMEM upon exceeding
              this limit.  Also automatic stack expansion will fail (and
              generate a SIGSEGV that kills the process if no alternate
              stack has been made available via sigaltstack(2)).  Since the
              value is a long, on machines with a 32-bit long either this
              limit is at most 2 GiB, or this resource is unlimited.

[l-arnold]

The whole TKL Jessie, by default now, is 64 Bit. Not sure if it matters then. Strange though that it would be "unlimited".

Anyway, maybe we should also reference some other implementations or see what they say in the Odoo Help forums on the subject. Seems like it is still an Odoo Issue inside.

[DocCyblade]

After reading this http://www.javacodegeeks.com/2014/09/using-rlimit-and-why-you-should.html

It seems this limit is to keep Odoo from crashing the whole system. I see this as being used in multi-hosting sites where they don't want a single Odoo instance running away and crashing everything else.

So even knowing what I know now, this limit is to keep the server from crashing from a run away process. (Think memory leak) With that said, we know that we need at least 1.3GB for the pdf module to render PDF files. I see this number starting at say 1.5GB for systems with 512MB ram, and going to 2GB for systems with 1GB of ram (i assume they have a swap file at least as big as the physical memory) have 2GB. If the system is 32bit the max is going to be 2GB. This seems to be a safe limit. Anything over 2GB seem excessive however if you had a very large server 16GB of ram and a 16GB swap file you could have a process that consumes up to 32GB of real/swap. I reality if a process is taking up that much there is a problem.

So I say a 2GB should cover VM with 1GB. Again the reason for this limit is to keep a memory leak or bad process to lockup a server

@JedMeister am I on base here? I just learned all this tonight and my head is going to explode !

[JedMeister]

@l-arnold

If you notice, this limit is maxed out on a 32bit server since it's a long number. A 64bit server has no limit.

Strange though that it would be "unlimited".

FWIW 64 bit is not really "unlimited"; but it is such a massively big number that it may as well be...

This is to do with addressable memory space: Using a 32 bit memory register (of byte addressable memory): 2^32 bytes (4GB) is the most that can be addressed. That is excluding PAE trickery - which allows extension of the addressable memory space to support of a 36bit physical address register i.e. 2^36 = 64GB.

Using 64 bit memory address space is a crazy big amount! 2^64 bytes = ~16.8 million TB!!! :scream: Obviously not a practical amount of memory...

@DocCyblade TBH I'm not sure... 2GB is probably a reasonable starting point and let end users tweak as needs be.

FWIW it's highly unlikely that Odoo (or any other process) would "take the whole system down" when run on Linux. This is because the Linux kernel has a feature known as OOM killer. When the system runs out of virtual memory space it will sacrifice the process (or processes) that are using the most RAM. This would mostly likely be the runaway Odoo worker... Having said that; if another (legitimate) process was using lots of RAM that may be killed too (or instead of?) so it may result in some system instability...

[DocCyblade]

Some good things found out in this issues discussion, however regarding the issues subject I think we covered this by setting the workers to 0.