Open CaptnMorgan833 opened 3 years ago
memsharded
commented
3 years ago Hi @CaptnMorgan833
You mean that you have asset files in the template? That is interesting, why would you be including the same image files in the conan package recipe files, so often to make it a template?
I don't find anything specific why an image will not be copied, let me have a look and try to reproduce with a unittest.
CaptnMorgan833
commented
3 years ago Hey @memsharded
Yes, exactly. I have a logo image, which I want to place inside my /doc folder to have the logo in the doxygen generated documentation.
memsharded
commented
3 years ago I can't reproduce, it seems to be working fine, this is the test unit I wrote:
def test_template_image_files(self):
client = TestClient()
template_dir = "templates/command/new/t_dir"
save(os.path.join(client.cache_folder, template_dir + "/myimage.png"), "$&()")
client.run("new hello/0.1 --template=t_dir")
myimage = client.load("myimage.png")
assert myimage == "$&()"Could you please double check it? Any other hint to reproduce the issue?
CaptnMorgan833
commented
3 years ago Hmm that is strange...
This is the log I get:
C:\FAS\conan_test\gesdgdsgf>conan new myTestmodule/0.0.1 -m basis_library Traceback (most recent call last): File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\conans\client\command.py", line 2115, in run method(args[0][1:]) File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\conans\client\command.py", line 206, in new self._conan.new(args.name, header=args.header, pure_c=args.pure_c, test=args.test, File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\conans\client\conan_api.py", line 94, in wrapper return f(api, *args, *kwargs) File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\conans\client\conan_api.py", line 257, in new files = cmd_new(name, header=header, pure_c=pure_c, test=test, File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\conans\client\cmd\new.py", line 402, in cmd_new files = _get_files_from_template_dir(template_dir=template, File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\conans\client\cmd\new.py", line 314, in _get_files_from_template_dir rendered_file = _render_template(load(f_path), name=name, version=version, File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\conans\client\cmd\new.py", line 294, in _render_template t = Template(text, keep_trailing_newline=True) File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\jinja2\environment.py", line 1031, in new return env.from_string(source, template_class=cls) File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\jinja2\environment.py", line 941, in from_string return cls.from_code(self, self.compile(source), globals, None) File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\jinja2\environment.py", line 638, in compile self.handle_exception(source=source_hint) File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\jinja2\environment.py", line 832, in handle_exception reraise(rewrite_traceback_stack(source=source)) File "C:\Users\myUser\AppData\Local\Programs\Python\Python38\lib\site-packages\jinja2_compat.py", line 28, in reraise raise value.with_traceback(tb) File "
", line 1436, in template jinja2.exceptions.TemplateSyntaxError: unexpected '}' ERROR: unexpected '}'
After removing the .png file it works fine.
CaptnMorgan833
commented
3 years ago Im using conan 1.33.0 btw. May this be a problem?
memsharded
commented
3 years ago Oh, that could be it, indeed, the template feature improved recentely, the directory case was not handled before. I'd suggest trying to upgrade.
CaptnMorgan833
commented
3 years ago Ok I tried it with version 1.38.0 , but I ran into the same problem unfortunately :(
CaptnMorgan833
commented
3 years ago I will let one of my colleagues try to reproduce the problem. Let's see if they got the same issue.
memsharded
commented
3 years ago Oh, I have an idea. Maybe the image files are being parsed and tried to apply the template! I will update my test for this
memsharded
commented
3 years ago No, no success, I cannot make the test fail, even using characters like {}. Can you please share your jinja2 version? Doing a pip list or similar (I am testing with Jinja2 2.11.3)
CaptnMorgan833
commented
3 years ago Got the same version: Jinja2 2.11.3
CaptnMorgan833
commented
3 years ago That is a really weird issue. I will try it together with another colleague on Monday.
I will give feedback :)
Thanks for your quick response, have a nice weekend!
CaptnMorgan833
commented
3 years ago Hey James,
My colleague tried it and could reproduce the error with a different png Image.
It seems like it depends on the specific image whether the copying is successful or not.
Jinja2 parses all these images and it seems to result in one of these three outcomes (depending on the parsed image data):
Does it make sense to each try it with the opposing iamges every one of us used? :)
memsharded
commented
3 years ago Ok, managed to reproduce it, trying different binary patterns.
I am working to try a fix, but it happens that is not that evident, to differentiate which files should be tried to apply the template, and which files shouldn't. We could start using file extension and guessing, maybe exclude some known binary formats, but eventually there will be someone with a file without extension that they don't want to be templatized.
We could use a file inside the template to list the files to be templatized or the files to exclude.
In any case, I was wondering, because maybe adding that logo .png to all the package sources might not be the best approach, as you will be reproducing, storing and transferring the same file for every package, as part of the source of the package. It seems that you want to have that logo as part of the final binary package (the docs), so why not getting the logos (and any other files used to generate those company branded docs) as part of the build process? The idea is that the Conan recipes, the smallest in size, the better, as they are transferred many more times than the binaries.
memsharded
commented
3 years ago https://github.com/conan-io/conan/pull/9323 would be reproducing the issue, with a dirty poc of a fix based on try-except to avoid jinja2 errors
CaptnMorgan833
commented
3 years ago Hey thanks. I will try that commit on my machine.
Yeah I thought about something like a ".gitignore" kind of file to exclude filetypes I do not want to be parsed.
And yes I agree having the same binary file in the sources is not the best approach. But for now it seems like the simplest solution to make it work. But in the future it is prefered to make this a part of the build process, just as you said.
Anyways, thank you for your support so far!
CaptnMorgan833
commented
3 years ago Hey James, I tried your commit with my logo.png file, but I get the following error, when using the template:
[...]\new.py", line 310, in _get_files_from_template_dir content = content.decode("utf-8") UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte
ERROR: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte
Without the image it works fine as usual.
memsharded
commented
3 years ago Yep, definitely the "gitignore" kind solution would be the more robust, no magic or autodetection, just declare which files shouldn't be applied the template. That would be a fairly easy feature.
memsharded
commented
3 years ago Yes, I forgot about the potential Unicode error. I have added a capture for that exception too, please try again. Thanks!
CaptnMorgan833
commented
3 years ago I'll try.
CaptnMorgan833
commented
3 years ago Now it works! Nice one!
memsharded
commented
2 years ago Functionality of https://github.com/conan-io/conan/pull/9323 has been moved to 2.0 to https://github.com/conan-io/conan/pull/10255, because this never moved forward in 1.X, as it didn't feel right. In that other PR, the exclusion is explicit, with a file that defines which patterns to be excluded.
I still thing that the way to go is to have those binary assets somewhere in the build, but I hope this helps, even if it needs to wait until 2.0 (now we are in alpha).
Hello,
I tried the nice template feature with my own template ("conan new myModuie/1.0.0 -m"). It works fine so far, except for the images I stored in the template. No matter which file type (png, jpg, bmp) the copied version cannot be displayed. Is this a Jinja issue or a conan issue?
Thanks!