Fails to detect that the JIT cache is stale and re-JIT

[gfxstrand]

We use mako heavily for various things in the Piglit 3D driver test suite. I ran into an issue this week when ping-ponging back and forth between two versions of piglit where mako fails to recognize that the data in its JIT cache in /tmp is stale and re-JIT when I switch versions. Whichever version I run first JITs and then, when I run the other version, it doesn't re-JIT but instead tries to use the python from the first one. This results in some rather esoteric crashes.

I'm running the python3-mako-1.1.0 package on Fedora 31

[zzzeek]

I'm not familar with a JIT cache in Python. are you referring to the generated .py files Mako creates, or .pyc, or something else entirely?

[gfxstrand]

I'm referring to the python files that mako generates and stashes in /tmp/<username>/python-<version>/mako-<version> Sorry if I'm not quite using the right terminology.

[zzzeek]

those are .py files that don't change unless the template does, and if you are referring to the .pyc files, the Python interpreter should be responsible for making sure those are against the right binary. it's not clear what piglet has to do with this. if you are replacing mako template files with new ones, in order for them to recompile, the timestamp of the .mako file has to be newer than that of the .py file, which is how it checks that a new .py must be generated.

[gfxstrand]

Yes, I'm talking about the .py files. I have two versions of the same application in different directories (the fact that it's piglit is immaterial) and am trying to use both of them simultaneously or switch back and forth between them. Because mako only uses timestamps and a relative file path to check for the validity of it's generated .py files, it doesn't actually generate two versions, one for each application, or re-generate on each run. When I run the other version, it assumes (because the timestamps haven't changed) that generated .py files are up-to-date and runs them without question. Because the two versions of the software are different, I get errors with undefined values when the template from one is used with the other.

[zzzeek]

it doesn't generate the .py file unless you are setting module_directory to something, which is optional. also it's an absolute path: https://docs.makotemplates.org/en/latest/usage.html#using-file-based-templates

if blazing speed is not a requirement here I would just not use module_directory, but otherwise take some identifying feature of each version and add that to the path used by your module_directory setting.

Or you can set the filename for every template explicitly using Template()->module_filename or TemplateLookup()->modulename_callable.

OR, I can't even believe there are this many options (I haven't worked on Mako in years), you can make module_writer() callable where you can write the whole .py file yourself. this definitely should not be needed, adding a version identifier to your module_directory should be enough.

these options are all at https://docs.makotemplates.org/en/latest/usage.html#mako.template.Template

[gfxstrand]

Thanks! Looking at the piglit code it looks like it's our fault. We do set a module_directory but don't make it unique enough. And here I was assuming (because I don't know mako all that well) that it was mako that was coming up with the temporary directory. :frowning: Sorry for the noise!