Data manager writes in unexpected .loc file

[abretaud]

Describe the bug Having a problem with 2 data managers, I think it's the same cause.

First one:

• a loc file (for data table ncbi_taxonomy) from cvmfs referenced in tool_data_table_config_path
• a data manager for the same data table, with a loc file referenced in shed_tool_data_table_config => When I run the data manager, Galaxy tries to write to the cvmfs loc file, and fails with a permission denied error :scream:

Second one:

• a loc file (for data table frogs_db) referenced in tool_data_table_config_path
• a data manager for the same data table, with a loc file referenced in shed_tool_data_table_config => When I run the data manager, Galaxy write to the first loc file instead of the second one. As the first loc file is managed by ansible, it gets rewritten on next CI run => the db disappears mysteriously :scream:

Galaxy Version and/or server at which you observed the bug Galaxy Version: 23.1 Commit: a3c14eaa

Browser and Operating System Operating System: Linux Browser: Firefox

To Reproduce See description above

Expected behavior I would expect that Galaxy write to the same loc file as the data manager which was executed, but for some reason it does not.

[bernt-matthias]

For the first problem you probably need to set galaxy_data_manager_data_path in the Galaxy config yaml.

For the second problem. Could you detail the exact configurations of the mentioned loc files and config files? Maybe also the docs that we may add soon-ish could help https://github.com/galaxyproject/galaxy/pull/17585?

To my understanding the data manager should write to the loc file in a versioned directory and you ansible role to a central one (e.g. in tool-data/).

[abretaud]

For the first problem you probably need to set galaxy_data_manager_data_path in the Galaxy config yaml.

Hmm I think this options is used by Galaxy to place the data itself? Here the data is placed in the good location, but Galaxy tries to write in a read-only .loc file on cvmfs

For the second problem. Could you detail the exact configurations of the mentioned loc files and config files? Maybe also the docs that we may add soon-ish could help #17585?

In galaxy.yml:

    shed_tool_data_table_config: /blabla/galaxy/mutable-config/shed_tool_data_table_conf.xml
    tool_data_table_config_path: /cvmfs/data.galaxyproject.org/byhand/location/tool_data_table_conf.xml,/cvmfs/data.galaxyproject.org/managed/location/tool_data_table_conf.xml,/blabla/galaxy/config/tool_data_table_conf_local.xml

In /blabla/galaxy/config/tool_data_table_conf_local.xml:

    <table name="frogs_db" comment_char="#">
        <columns>value, name, path</columns>
        <file path="/shared/ifbstor1/galaxy/mutable-config/tool-data//frogs_db.loc" />
    </table>

In /blabla/galaxy/mutable-config/shed_tool_data_table_conf.xml (greped):

    <table comment_char="#" name="frogs_db">
        <columns>value, name, path</columns>
        <file path="/blabla/galaxy/mutable-config/tool-data/toolshed.g2.bx.psu.edu/repos/frogs/frogs/094a2469204d/frogs_db.loc"/>
        <tool_shed_repository>
            <tool_shed>toolshed.g2.bx.psu.edu</tool_shed>
            <repository_name>frogs</repository_name>
            <repository_owner>frogs</repository_owner>
            <installed_changeset_revision>094a2469204d</installed_changeset_revision>
        </tool_shed_repository>
    </table>
--
    <table comment_char="#" name="frogs_db">
        <columns>value, name, path</columns>
        <file path="/blabla/galaxy/mutable-config/tool-data/toolshed.g2.bx.psu.edu/repos/frogs/frogs/8edcbafb3b4e/frogs_db.loc"/>
        <tool_shed_repository>
            <tool_shed>toolshed.g2.bx.psu.edu</tool_shed>
            <repository_name>frogs</repository_name>
            <repository_owner>frogs</repository_owner>
            <installed_changeset_revision>8edcbafb3b4e</installed_changeset_revision>
        </tool_shed_repository>
    </table>
--
    <table comment_char="#" name="frogs_db">
        <columns>value, name, path</columns>
        <file path="/blabla/galaxy/mutable-config/tool-data/toolshed.g2.bx.psu.edu/repos/frogs/data_manager_frogs/7403d6c4f510/frogs_db.loc"/>
        <tool_shed_repository>
            <tool_shed>toolshed.g2.bx.psu.edu</tool_shed>
            <repository_name>data_manager_frogs</repository_name>
            <repository_owner>frogs</repository_owner>
            <installed_changeset_revision>7403d6c4f510</installed_changeset_revision>
        </tool_shed_repository>
    </table>
--
    <table comment_char="#" name="frogs_db">
        <columns>value, name, path</columns>
        <file path="/blabla/galaxy/mutable-config/tool-data/toolshed.g2.bx.psu.edu/repos/frogs/frogs/445b04a65ed8/frogs_db.loc"/>
        <tool_shed_repository>
            <tool_shed>toolshed.g2.bx.psu.edu</tool_shed>
            <repository_name>frogs</repository_name>
            <repository_owner>frogs</repository_owner>
            <installed_changeset_revision>445b04a65ed8</installed_changeset_revision>
        </tool_shed_repository>
    </table>
--
    <table comment_char="#" name="frogs_db">
        <columns>value, name, path</columns>
        <file path="/blabla/galaxy/mutable-config/tool-data/toolshed.g2.bx.psu.edu/repos/frogs/frogs/37e6f0c959bb/frogs_db.loc"/>
        <tool_shed_repository>
            <tool_shed>toolshed.g2.bx.psu.edu</tool_shed>
            <repository_name>frogs</repository_name>
            <repository_owner>frogs</repository_owner>
            <installed_changeset_revision>37e6f0c959bb</installed_changeset_revision>
        </tool_shed_repository>
    </table>
--
    <table comment_char="#" name="frogs_db">
        <columns>value, name, path</columns>
        <file path="/blabla/galaxy/mutable-config/tool-data/toolshed.g2.bx.psu.edu/repos/frogs/frogs/0966b9635d76/frogs_db.loc"/>
        <tool_shed_repository>
            <tool_shed>toolshed.g2.bx.psu.edu</tool_shed>
            <repository_name>frogs</repository_name>
            <repository_owner>frogs</repository_owner>
            <installed_changeset_revision>0966b9635d76</installed_changeset_revision>
        </tool_shed_repository>
    </table>

To my understanding the data manager should write to the loc file in a versioned directory and you ansible role to a central one (e.g. in tool-data/).

That's what I thought too, but apparently for these too cases Galaxy doesn't write to the versioned directory

[bebatut]

We still have the same issue: we have to run the DM every month. Would there be a way to fix this?