[bug] metadata upload md5 files

[chfuerst]

Describe the bug

OS: Ubuntu22.04 Conan version 2.7.0

I am experimenting around with recipe metadata.

User story: As a C++ developer, i want to publish / upload Doxygen documentation as recipe metadata to my package.

My C++ project contains Doxygen documentation, this is, inline C++ comments, which are extracted by Doxygen, to generate html-Documentation. I want to store the derived html directory in the recipe metadata. The list of files include:

and i use the conanfile copy command to package my recipe metadata. I upload every file succesful except for the *.md5 files, which are either not accepted by artifactory, or ignored by the conan upload command.

Renamed the files to *.md5.1 -> could upload immediately.

Is this some verification algorithm from conan that interpretes *.md5 in way that it verifies checksum but does not upload it by default? As i learned about md5 files

• https://lists.debian.org/debian-lint-maint/2014/06/msg00021.html
• https://marc.info/?l=doxygen-users&m=111350644016377

Are md5 files interpreted as checksums and hence not uploaded? Encountered the same issue for repository type "Conan" and "Generic" in artifactory.

How to reproduce it

Upload *.md5 data as metadata.

[memsharded]

Hi @chfuerst

Thanks for your report. I have tried to reproduce with a unittest, and it is working fine, so it seems there is nothing in the Conan client that filters md5 files. This is the test:

    def test_upload_md5(self, create_conan_pkg):
        c, pid = create_conan_pkg

        # Add some metadata
        self.save_metadata_file(c, "pkg/0.1", "myfile.md5")
        self.save_metadata_file(c, f"pkg/0.1:{pid}", "mybin.md5")
        # Now upload everything
        c.run("upload * -c -r=default")
        assert "pkg/0.1: Recipe metadata: 1 files" in c.out
        assert "pkg/0.1:da39a3ee5e6b4b0d3255bfef95601890afd80709: Package metadata: 1 files" in c.out

        # Add new files to the metadata
        self.save_metadata_file(c, "pkg/0.1", "other.md5")
        self.save_metadata_file(c, f"pkg/0.1:{pid}", "otherbin.md5")
        # Upload the metadata, even if the revisions exist in the server
        # adding the new metadata logs files
        c.run("upload * -c -r=default --metadata=*")
        assert "pkg/0.1: Recipe metadata: 2 files" in c.out
        assert "pkg/0.1:da39a3ee5e6b4b0d3255bfef95601890afd80709: Package metadata: 2 files" in c.out

        # Forcing the download of the metadata of cache-existing things with the "download" command
        c.run("download pkg/0.1 -r=default --metadata=*")
        c.run(f"cache path pkg/0.1 --folder=metadata")
        metadata_path = str(c.stdout).strip()
        c.run(f"cache path pkg/0.1:{pid} --folder=metadata")
        pkg_metadata_path = str(c.stdout).strip()
        for f in "logs/myfile.md5", "logs/other.md5":
            assert os.path.isfile(os.path.join(metadata_path, f))
        for f in "logs/mybin.md5", "logs/otherbin.md5":
            assert os.path.isfile(os.path.join(pkg_metadata_path, f))

I will do a quick check with Artifactory server.

[memsharded]

It seems that is an Artifactory thing, I am also getting:

WARN: network: 404 Client Error: for url: http://localhost:8081/artifactory/api/conan/conan/v2/conans/mymetadata/0.1/_/_/revisions/25597a97c662d12f711a91e650dbfdde/files/metadata/myfile.md5

Investigating, I have seen that this is Artifactory specific behavior see for example: https://stackoverflow.com/questions/23478236/cannot-deploy-file-md5-target-file-to-set-checksum-on-doesnt-exist-artifact

Because checksum extension files have a special meaning have a special meaning for Artifactory.

[memsharded]

I have managed to make it work by the following:

• The checksum files must be uploaded after the files they are providing the checksum for
• That happens if the checksum file is correctly named, as files are naturally (alphabetically sorted)
• So if the file is named myfile.txt the correct checksum is myfile.txt.md5. This prevents issues, if there is another myfile.exe file, for example.

So my recommendation would be:

• The md5 files are mostly an implementation detail of doxygen, used to speed up the generation of pages. They are not needed in the final result, so the recommendation would be to clean them after the pages have been generated and are going to be uploaded as metadata

[chfuerst]

Thank you @memsharded - i am wondering whether the upload order (i.e. the alphabetical order) you mentioned in point 1 is covered by conan: Generally, i have understood, that for each file a corresponding *.md5 file is in alphabetical order right after the file it corresponds to; However the "sort-step" is essential;

Summarizing, Artifactory has a restriction that one can not upload md5-files without a corresponding file, if i want to upload

.md5 i have to ensure that is also available and uploaded first. Thank you, Christoph

[memsharded]

Yes, files uploads are sorted, so file.txt.md5 will be uploaded always after file.txt. But the same cannot be said for file.md5 and file.txt, the md5 will be uploaded first and it will fail.

Still it seems that removing the .md5 files before the upload is the way to go.

[chfuerst]

Understood. Thank you for clarification. Not sure, whether this is the "expected" behavior from Artifactory, but since every file in Artifactory has a corresponding md5-hash, one might does not need to upload single md5-files.

Yes, i have deleted the *.md5 files from my generated doxygen documentation, i am fine :-).

Thank you, you can close the ticket.

[memsharded]

Not sure, whether this is the "expected" behavior from Artifactory, but since every file in Artifactory has a corresponding md5-hash, one might does not need to upload single md5-files.

Yes, this is a security feature of Artifactory, that validates all checksum files to guarantee they are correct, but it needs them to be uploaded in the right order and with the right naming.

Thanks for the feedback!