Generalized linear mixed models

[maikherbig]

Currently only linear mixed models are available via the analysis tab. Generalized linear mixed models help to address the lognormality of deformation. To implement this, probably a dropdown menu is sufficient which allows the user to select between 'linear mixed model' and 'generalized linear mixed model' (and maybe more in the future???). linmixmod.py could be easily adjusted to deal with both cases, or I write a new function (genmixmod.py).

[paulmueller]

To avoid redundant code, it would be best to implement this functionality in lin_mix_mod.py. Please create a pull-request with the necessary changes. I can add the dropdown menu afterwards.

[paulmueller]

The pull request was #135 and was merged into the separate https://github.com/ZELLMECHANIK-DRESDEN/ShapeOut/tree/glmm branch. I repaired the new version of lin_mix_mod.py. However, there are a few things missing until I am willing to merge into develop:

• [x] a test function for the generalized linear mixed effects model (glmm)
• [x] an explanation or fix for the fact that this case: https://github.com/ZELLMECHANIK-DRESDEN/ShapeOut/blob/glmm/shapeout/lin_mix_mod.py#L198 will always have the Head_string "LINEAR...", regardless of the value of model (which could also be "glmm").

[katpla]

Hi,

I am not starting a new topic as I think it may belong here. Hitting Apply button in Analysis tab gives me an error message .......... line 289, in linmixmod UnicodeDecodeError: 'ascii' codex cant decode byte 0xf6 in position 28: ordinal not in range(128) it's true for both lmm and glmm i tried both 32 and 64bit versions of latest release shapeout

[paulmueller]

I assume you are talking about the latest build from the develop branch - Glmm is not yet implemented in the latest release. https://ci.appveyor.com/project/paulmueller/shapeout/build/1.0.567

Yes, this belongs here. I need to figure out how to debug this.

[katpla]

yes, this is the one I have right now. MaikH88 has tried and it is running on his machine so he asked me if it works for me.

[paulmueller]

Are you two using the same Version of Windows? I don't know why it works for him and not for you.

[maikherbig]

Me: Win7 64bit and used ShapeOut 32bit (ShapeOut_0.7.1.dev17_win_32bit_setup.exe)

[katpla]

Win7 Professional 64bit. currently running 64bit shapeout (ShapeOut_0.7.1.dev17_win_64bit_setup)

changing to 32bitshapeout did not help

[paulmueller]

If MaikH88 is using a German-Language Windows and katpla an English-language Windows: katpla, could you please try setting your system language to German?

[katpla]

First of all, maybe I should mention that the lmm used to work for me in shapeout before the gmm were introduced (before version 7.1). So that may point against the language hypothesis

Also I guess it is not enough to change the display language right? I am afraid that to change system language will be more difficult as it may require our IT department to be involved (for some reason I can not have administrator privileges on my computer). MaikHH if you are an administrator on your machine can you change your system language to English and see if it breaks the lmm/gmm?

[paulmueller]

I ran into UTF-8 problems myself when I tested the glmm functionality. Apparently, "R" spits out UTF-8 characters for glmm but not for lmm. I then changed everything to UTF-8 and it worked again. Unfortunately, that broke things for you. I will have to dig deeper to find a workaround.

[katpla]

I was advised to change my regional settings to English (USA). then after restarting it seems it is working..... for lmm, but not for glm

[paulmueller]

Advised by whom and what was the previous setting?

[katpla]

Geo, and it was German. it fixed the problem for lmm. gmm gives me a different type of error. Will explore a bit to see what is up and then update you

[maikherbig]

I see in the output (after hitting "Apply") that all '\n' were removed from the string. Therefore this output is not barely legible.

[paulmueller]

@MaikH88 that's probably just a "\r" missing.

[maikherbig]

I just checked the code and its fine. It must be removed at some later point

[katpla]

I managed to perform the analysis on gmm by reducing the no of datapoints. So what is going on after changing the settings to English USA (as an example I did analysis on a dataset, choosing Area as axis to analyze):

lmm - gives a decent result: 73um average - which fits with my cell sizes glmm - gives some insane result with average cell size of: 4.29 which makes no sense whatsoever.

[maikherbig]

glmm results are in logspace At the end of the file you find the estimate and the effect transformed back

[maikherbig]

exp(4.29) = 72.996 :) nice

[katpla]

Ok Maik, thanks I see this now. so it is working on my prefilterd data (1.05 filter). If I use unfiltered data the glmm works for deformation, area etc, but not for emodulus - which I guess may have something to do with the problem you mentioned that some ns are missing. Is that correct?

[katpla]

sorry, by emodulus i ment userdef1

[maikherbig]

you are right - userdef1 can be empty for some cells I just tried glmm and lmm with some data that contains nan (numpy.nan) and it worked fine. So in principle these packages are robust for missing data. Probably you dont have it numpy.nan's anymore since you do some converting (Origin?)

[katpla]

I put 0s in all empty cells

[maikherbig]

Zeros are causing a problem because of the log-link function. The log(0) is not defined.

[katpla]

Maik, does it has something to do with the fact that you changed your computer to a new one recently? was the lmm written on a machine with different settings? For the initial problem - the one that got fixed by changing the regional settings?

[katpla]

Ok: so there were two problems that got solved: problem 1 - was fixed by the change to a diferent regional settings - has to do with the comas and dots probably problem 2 is mathematical, due to the fact that I filled my empty cells with '0'. - this one is specific to my data and I can work with it easily by choosing anything but 0s. Anyhow I filter those events out as they are bad cells with no image corresponding etc.

I guess it is working now. Thanks Guys!

[maikherbig]

Awesome work! Cheers :)

[paulmueller]

Problem 1 still persists and has nothing to do with commas or dots. It is caused by a weird interplay b/w Windows, R, and pyper.

[paulmueller]

I am moving problem 1 to a new issue: #142