search

nextflow-io / nextflow

A DSL for data-driven computational pipelines
http://nextflow.io
Apache License 2.0
2.73k stars 626 forks source link

nf fails to pull from gitlab when repo is under a subgroup #1363

Closed af8 closed 4 years ago

af8 commented 4 years ago

Bug report

I would like to use something like : nextflow run -hub gitlab auragen/dev/pipelines/testlab-nf ... to run private nextflow repo testlab-nf.

First I can actually run it successfully if I use the full http address : 

> nextflow run https://gitlab.com/auragen/dev/pipelines/testlab-nf
...
...
> head .nextflow.log
Nov-08 12:46:32.539 [main] DEBUG nextflow.cli.Launcher - $> nextflow run 'https://gitlab.com/auragen/dev/pipelines/testlab-nf'
Nov-08 12:46:32.740 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 19.01.0
Nov-08 12:46:33.718 [main] DEBUG nextflow.scm.AssetManager - Repository URL: https://gitlab.com/auragen/dev/pipelines/testlab-nf; Project: auragen/dev/pipelines/testlab-nf; Hub provider: gitlab
Nov-08 12:46:33.727 [main] INFO  nextflow.cli.CmdRun - Pulling auragen/dev/pipelines/testlab-nf ...
Nov-08 12:46:33.733 [main] DEBUG nextflow.scm.RepositoryProvider - Request [credentials af8:********************] -> https://gitlab.com/api/v4/projects/auragen%2Fdev%2Fpipelines%2Ftestlab-nf
Nov-08 12:46:35.281 [main] DEBUG nextflow.scm.RepositoryProvider - Request [credentials af8:********************] -> https://gitlab.com/api/v4/projects/auragen%2Fdev%2Fpipelines%2Ftestlab-nf/repository/files/nextflow.config?ref=master
Nov-08 12:46:35.788 [main] DEBUG nextflow.scm.RepositoryProvider - Request [credentials af8:********************] -> https://gitlab.com/api/v4/projects/auragen%2Fdev%2Fpipelines%2Ftestlab-nf/repository/files/main.nf?ref=master
Nov-08 12:46:36.002 [main] DEBUG nextflow.scm.AssetManager - Pulling auragen/dev/pipelines/testlab-nf -- Using remote clone url: https://gitlab.com/auragen/dev/pipelines/testlab-nf.git
Nov-08 12:46:39.091 [main] INFO  nextflow.cli.CmdRun -  downloaded from https://gitlab.com/auragen/dev/pipelines/testlab-nf.git
Nov-08 12:46:39.140 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/anthony/.nextflow/assets/auragen/dev/pipelines/testlab-nf/.git/config; branch: master; remote: origin; url: https://gitlab.com/auragen/dev/pipelines/testlab-nf.git

But if I try : 

> nextflow run -hub gitlab auragen/dev/pipelines/testlab-nf
N E X T F L O W  ~  version 19.01.0
Not a valid project name: auragen/dev/pipelines/testlab-nf

I get an error because the code only tolerates two levels of directories, from AssetManager.groovy : 

   boolean isValidProjectName( String projectName ) {
        projectName =~~ /.+\/.+/
    }

To try to escape that problem, I have created a scm configuration file containing this : 

    gitlab {
        user = 'af8'
        password = 'xxxxxxxxxxxx'
        token = 'xxxxxxxxxxxx'
    }
    gitlab_au {
        server = 'https://gitlab.com/auragen/dev'
        platform = 'gitlab'
        user = 'af8'
        password = 'xxxxxxxxxxxx'
        token = 'xxxxxxxxxxxx'
    }

And then I run : 

> nextflow run -hub gitlab_au pipelines/testlab-nf
N E X T F L O W  ~  version 19.01.0
Pulling pipelines/testlab-nf ...
WARN: Cannot read project manifest -- Cause: Unable to determine the current character, it is not a string, number, array, or object

The current character read is '<' with an int value of 60
Unable to determine the current character, it is not a string, number, array, or object
line number 1
index number 0
<!DOCTYPE html>
^
ERROR ~ Unable to determine the current character, it is not a string, number, array, or object

The current character read is '<' with an int value of 60
Unable to determine the current character, it is not a string, number, array, or object
line number 1
index number 0
<!DOCTYPE html>
^

 -- Check '.nextflow.log' file for details

> head .nextflow.log
Nov-08 13:10:27.274 [main] DEBUG nextflow.cli.Launcher - $> nextflow run -hub gitlab_au pipelines/testlab-nf
Nov-08 13:10:27.494 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 19.01.0
Nov-08 13:10:28.525 [main] INFO  nextflow.cli.CmdRun - Pulling pipelines/testlab-nf ...
Nov-08 13:10:28.542 [main] DEBUG nextflow.scm.RepositoryProvider - Request [credentials af8:********************] -> https://gitlab.com/auragen/dev/api/v4/projects/pipelines%2Ftestlab-nf
Nov-08 13:10:30.532 [main] WARN  nextflow.scm.AssetManager - Cannot read project manifest -- Cause: Unable to determine the current character, it is not a string, number, array, or object

The current character read is '<' with an int value of 60
Unable to determine the current character, it is not a string, number, array, or object
line number 1
index number 0

You see here that the problem is that the end point is not correctly built => https://gitlab.com/auragen/dev/api/v4/projects/pipelines%2Ftestlab-nf

So I did not find an easy solution to allow nextflow run to work with several subdirs when pulling from gitlab.

I think that relaxing a bit the condition in isValidProjectName to authorize several levels could be the simple way to go ?

Thanks, Anthony

Environment

af8 commented 4 years ago

Or perhaps @pditommaso a new variable (group or path or subpath ...) can be created in scm gitlab config : 

providers {
    gitlab_au_v2 {
        server = 'https://gitlab.com'
        group = 'auragen/dev'
        platform = 'gitlab'
        user = 'af8'
        password = 'xxxxxxxxxxxx'
        token = 'xxxxxxxxxxxx'
    }
}

Then group value can be used to correctly build the endpoint path.

In this case I would run : 

nextflow run -hub gitlab_au_v2 pipelines/testlab-nf
stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Midnighter commented 3 years ago

I just ran into this problem as well. Can the issue be re-opened please? I wanted to run

nextflow pull https://gitlab.com/<org>/<project>/<repo>

and then get an error

Remote resource not found: https://gitlab.com/api/v4/projects/<org>%2F<project>%2F<repo>
abhi18av commented 3 years ago

Hi @Midnighter ,

Could you please make sure that the both password and token fields are specified correctly?

If this doesn't work, could you please tell us which version of Nextflow are you using. Output of nextflow info would be ideal :)

Midnighter commented 3 years ago

Hi,

Thanks for the message. I have changed my ~/.nextflow/scm config to contain:

providers {
    gitlab {
        user = 'Midnighter'
        password = '<token>'
        token = '<token>'
    }   
}

and now it works 🎉

For future me: I used the same API access token in both fields.