Specifying output directory and other compilation options for all Coconut files in the project

[kxmh42]

Say I have two files. file1.coco:

import file2
file2.func()

file2.coco:

def func():
    pass

Now if I want to run file1.coco, e.g. with coconut-run -l file1.coco, the -l option only applies to file1.coco and I have no way of changing the compilation options for file2.coco, is that right?

Specifically, what I'm seeking is a method to automatically redirect all compiled Python files to a designated subdirectory like __pycache__. By doing so, it would help keep them organized and easy to remove or gitignore. That's similar to the rationale behind placing .pyc files separately in __pycache__ rather than mixing them with the .py files in a single directory.

Not sure if that's one issue, or two related ones.

PS. Thanks for your great work. Coconut is awesome!

[evhub]

Now if I want to run file1.coco, e.g. with coconut-run -l file1.coco, the -l option only applies to file1.coco and I have no way of changing the compilation options for file2.coco, is that right?

That command will only compile file1.coco, it won't touch file2.coco at all. If you want to automatically compile Coconut files on import, you have to use automatic compilation via either import coconut.api or coconut --site-install.

Currently, the compilation parameters used for automatic compilation are not adjustable, however. That could be added as a new feature here, but what compilation parameters are you looking to adjust? Automatic compilation already includes --target sys --line-numbers --keep-lines.

We could also definitely make automatic compilation default to some specific directory like __pycache__. In general, though, the problem with automatic compilation more generally is that it only works if you have Coconut installed, which means you have to distribute Coconut with your code. If you just compile from a source dir to a compiled dir and distribute the compile dir (e.g. as in https://github.com/evhub/bbopt), then you don't need to distribute Coconut with your code.

[kxmh42]

That command will only compile file1.coco, it won't touch file2.coco at all. If you want to automatically compile Coconut files on import, you have to use automatic compilation via either import coconut.api or coconut --site-install.

Really? It does compile file2.coco in my case. I didn't run coconut --site-install, I simply installed it with Poetry.

$ ls
file1.coco  file2.coco
$ cat file1.coco
import file2
file2.func()
$ cat file2.coco
def func():
    pass
$ coconut -v
Coconut: Version 3.0.2 running on Python 3.11.2 and Cython cPyparsing v2.4.7.2.1.1
$ coconut-run -l file1.coco
$ ls
__pycache__  file1.coco  file1.py  file2.coco  file2.py

Currently, the compilation parameters used for automatic compilation are not adjustable, however. That could be added as a new feature here, but what compilation parameters are you looking to adjust? Automatic compilation already includes --target sys --line-numbers --keep-lines.

I've just realised that maybe I don't really need to adjust these parameters when invoking coconut-run. The only thing that I'd really find useful would be to put the compiled files in some specific directory.

We could also definitely make automatic compilation default to some specific directory like __pycache__. In general, though, the problem with automatic compilation more generally is that it only works if you have Coconut installed, which means you have to distribute Coconut with your code. If you just compile from a source dir to a compiled dir and distribute the compile dir (e.g. as in https://github.com/evhub/bbopt), then you don't need to distribute Coconut with your code.

Well, anyone using my Python code has to install pip dependencies anyway, and Coconut is just another pip dependency, isn't it? I'm mostly doing research code, and people using my code need an easy way to modify it, so releasing it in source form seems natural. Sometimes it's preferable to distribute the compiled code and minimise the number of dependencies, but source-only distribution is an important use case IMHO. Sure, running code with coconut-run is a bit less efficient, but if I really cared about that kind of efficiency, I wouldn't use Python either.

[evhub]

Really? It does compile file2.coco in my case. I didn't run coconut --site-install, I simply installed it with Poetry.

You must be importing coconut.api somehow, perhaps transitively, in file1.coco.

[evhub]

Well, anyone using my Python code has to install pip dependencies anyway, and Coconut is just another pip dependency, isn't it? I'm mostly doing research code, and people using my code need an easy way to modify it, so releasing it in source form seems natural. Sometimes it's preferable to distribute the compiled code and minimise the number of dependencies, but source-only distribution is an important use case IMHO. Sure, running code with coconut-run is a bit less efficient, but if I really cared about that kind of efficiency, I wouldn't use Python either.

The other things that are tricky about this mechanism of distributing Coconut code are:

1. The "starting point" for your package has to be a .py file that imports coconut.api, which can make it a bit difficult for people to use your package easily. E.g. if you put it in __init__.py, but then the user tries to import package.module rather than just package, I think it'll skip the __init__.py and fail to find package.module (though maybe I should test that).
2. It requires the user to have a compatible version of Coconut installed locally. Maybe this is fine, since you can just pin the dependency, but it makes it harder to e.g. deprecate stuff if this is happening widely in the ecosystem.

Maybe another option that would resolve (1) could be to have a mechanism for allowing your package to automatically run coconut --site-install on pip install.

[kxmh42]

You must be importing coconut.api somehow, perhaps transitively, in file1.coco.

It looks like coconut.api is automatically imported by coconut-run when I run coconut-run file1.coco. And this is how it should work, IMHO. It is not imported when I run coconut file1.coco; python file1.py (but that was not my example).

[evhub]

Ah, yep, you're right, it is doing that. Automatic compilation is always enabled at the interpreter, and coconut-run is using the interpreter to execute the code it runs. That should actually be documented, though.

[kxmh42]

The other things that are tricky about this mechanism of distributing Coconut code are:

I agree, these are very good points. If something is a package, most users won't care about its source and will just use the API. It makes sense to distribute Coconut packages in their most efficient, compiled form, so that they can be easily imported from Python without much hassle or unnecessary dependencies.

But my use case is much more experimental: I might want to try something out, run coconut-run myscript.coco, then change something, run coconut-run myscript.coco again, etc. In this case, I don't want myscript.py to get in the way, I don't want to worry about it any more than I worry about __pycache__/*.pyc files.

And again, if I want to send my code to a colleague for them to adapt it to their needs, it makes no sense to give them the compiled version, because they will have to modify my code anyway. This is the way many interpreted scripting languages are used, and Coconut can be used this way too. I'm just asking to make the compilation process a little less visible.

[kxmh42]

BTW, one case that is unintuitive due to the fixed auto-compile options is when you run coconut-run -n file1.coco. Then file1.py does not get created, but file2.py does. No big deal for me, just pointing this out. Maybe the -n option won't be needed at all if auto-compiled files go into a cache directory? That would also resolve https://github.com/evhub/coconut/issues/314.

[evhub]

As of coconut-develop>=3.0.2-post_dev21, I've changed auto compilation such that parameters that affect the compiled code (so e.g. --no-tco or --no-wrap-types) will be passed along and used in auto compilation, though any parameters that just affect the operation of the compiler itself (so e.g. --no-write) will still be ignored. Doesn't fully address your use case, but does help in keeping auto compiled files synced up with the main file when using coconut-run.

[evhub]

As of coconut-develop>=3.0.2-post_dev23, both coconut-run and all auto compilation will now compile to a __coconut_cache__ directory by default (though note this feature only works on Python 3.4+).

To get the new behavior, pip uninstall coconut and pip install -U coconut-develop>=3.0.2-post_dev23.