nextflow-io / nextflow

A DSL for data-driven computational pipelines
http://nextflow.io
Apache License 2.0
2.61k stars 606 forks source link

Fix security vulnerabilities #5014

Closed bentsherman closed 1 month ago

bentsherman commented 1 month ago

Close #5001

This commit also needs to be backported: https://github.com/nextflow-io/nextflow/commit/a310c77704a9d1dc301dec004a1be1d905c33589

pditommaso commented 1 month ago

Fail 😢

pditommaso commented 1 month ago

Now failing https tests 👿

bentsherman commented 1 month ago

Lol I think this is because the nextflow docs have changed:

https://github.com/nextflow-io/nextflow/blob/2e8b8fdf64fef17b7212a65661bb5eaa96d9c581/modules/nf-httpfs/src/test/nextflow/file/http/HttpFilesTests.groovy#L103-L117

Condition not satisfied:

lines[0] == '<!DOCTYPE html>'
|    |   |
|    |   false
|    |   2115 differences (0% similarity)
|    |   <!DOCTYPE html>(<html lang="en"> <head><meta charset="UTF-8"><meta name="description" content="Astro description"><meta name="viewport" content="width=device-width"><link rel="icon" type="image/svg+xml" href="/favicon.svg"><meta name="generator" content="Astro v4.6.1"><title>A DSL for parallel and scalable computational pipelines | Nextflow</title><meta name="description" content="Nextflow enables scalable and reproducible scientific workflows using software containers. It allows the adaptation of pipelines written in the most common scripting languages."><meta property="og:title" content="A DSL for parallel and scalable computational pipelines | Nextflow"><meta property="og:description" content="Nextflow enables scalable and reproducible scientific workflows using software containers. It allows the adaptation of pipelines written in the most common scripting languages."><meta property="og:image" content="https://nextflow.io/img/share.png"><meta property="og:type" content="website"><meta data-react-helmet="true" name="twitter:card" content="summary_large_image"><meta data-react-helmet="true" name="twitter:creator" content="@nextflowio"><meta property="twitter:title" content="A DSL for parallel and scalable computational pipelines | Nextflow"><meta property="twitter:description" content="Nextflow enables scalable and reproducible scientific workflows using software containers. It allows the adaptation of pipelines written in the most common scripting languages."><meta property="twitter:image" content="https://nextflow.io/img/share.png"><!-- Bootstrap core CSS --><link href="/css/bootstrap.css" rel="stylesheet"><!-- Custom styles for this template --><link href="/css/color-styles.css" rel="stylesheet"><link href="/css/ui-elements.css" rel="stylesheet"><link href="/css/custom.css" rel="stylesheet"><!-- Resources --><link href="/css/animate.css" rel="stylesheet"><link href="/css/summit.css" rel="stylesheet"><link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet"><link href="/fonts/degular/degular.css" rel="stylesheet"><!-- Google Tag Manager --><script>)
|    |   <!DOCTYPE html>(---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------)
|    <!DOCTYPE html><html lang="en"> <head><meta charset="UTF-8"><meta name="description" content="Astro description"><meta name="viewport" content="width=device-width"><link rel="icon" type="image/svg+xml" href="/favicon.svg"><meta name="generator" content="Astro v4.6.1"><title>A DSL for parallel and scalable computational pipelines | Nextflow</title><meta name="description" content="Nextflow enables scalable and reproducible scientific workflows using software containers. It allows the adaptation of pipelines written in the most common scripting languages."><meta property="og:title" content="A DSL for parallel and scalable computational pipelines | Nextflow"><meta property="og:description" content="Nextflow enables scalable and reproducible scientific workflows using software containers. It allows the adaptation of pipelines written in the most common scripting languages."><meta property="og:image" content="https://nextflow.io/img/share.png"><meta property="og:type" content="website"><meta data-react-helmet="true" name="twitter:card" content="summary_large_image"><meta data-react-helmet="true" name="twitter:creator" content="@nextflowio"><meta property="twitter:title" content="A DSL for parallel and scalable computational pipelines | Nextflow"><meta property="twitter:description" content="Nextflow enables scalable and reproducible scientific workflows using software containers. It allows the adaptation of pipelines written in the most common scripting languages."><meta property="twitter:image" content="https://nextflow.io/img/share.png"><!-- Bootstrap core CSS --><link href="/css/bootstrap.css" rel="stylesheet"><!-- Custom styles for this template --><link href="/css/color-styles.css" rel="stylesheet"><link href="/css/ui-elements.css" rel="stylesheet"><link href="/css/custom.css" rel="stylesheet"><!-- Resources --><link href="/css/animate.css" rel="stylesheet"><link href="/css/summit.css" rel="stylesheet"><link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet"><link href="/fonts/degular/degular.css" rel="stylesheet"><!-- Google Tag Manager --><script>
[<!DOCTYPE html><html lang="en"> <head><meta charset="UTF-8"><meta name="description" content="Astro description"><meta name="viewport" content="width=device-width"><link rel="icon" type="image/svg+xml" href="/favicon.svg"><meta name="generator" content="Astro v4.6.1"><title>A DSL for parallel and scalable computational pipelines | Nextflow</title><meta name="description" content="Nextflow enables scalable and reproducible scientific workflows using software containers. It allows the adaptation of pipelines written in the most common scripting languages."><meta property="og:title" content="A DSL for parallel and scalable computational pipelines | Nextflow"><meta property="og:description" content="Nextflow enables scalable and reproducible scientific workflows using software containers. It allows the adaptation of pipelines written in the most common scripting languages."><meta property="og:image" content="https://nextflow.io/img/share.png"><meta property="og:type" content="website"><meta data-react-helmet="true" name="twitter:card" content="summary_large_image"><meta data-react-helmet="true" name="twitter:creator" content="@nextflowio"><meta property="twitter:title" content="A DSL for parallel and scalable computational pipelines | Nextflow"><meta property="twitter:description" content="Nextflow enables scalable and reproducible scientific workflows using software containers. It allows the adaptation of pipelines written in the most common scripting languages."><meta property="twitter:image" content="https://nextflow.io/img/share.png"><!-- Bootstrap core CSS --><link href="/css/bootstrap.css" rel="stylesheet"><!-- Custom styles for this template --><link href="/css/color-styles.css" rel="stylesheet"><link href="/css/ui-elements.css" rel="stylesheet"><link href="/css/custom.css" rel="stylesheet"><!-- Resources --><link href="/css/animate.css" rel="stylesheet"><link href="/css/summit.css" rel="stylesheet"><link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet"><link href="/fonts/degular/degular.css" rel="stylesheet"><!-- Google Tag Manager --><script>,       (function (w, d, s, l, i) {,         w[l] = w[l] || [];,         w[l].push({ "gtm.start": new Date().getTime(), event: "gtm.js" });,         var f = d.getElementsByTagName(s)[0],,           j = d.createElement(s),,           dl = l != "dataLayer" ? "&l=" + l : "";,         j.async = true;,         j.src = "https://www.googletagmanager.com/gtm.js?id=" + i + dl;,         f.parentNode.insertBefore(j, f);,       })(window, document, "script", "dataLayer", "GTM-TNCXSWG");,     </script><!-- End Google Tag Manager --></head> <body class="body-green"> <!-- Google Tag Manager (noscript) --> <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-TNCXSWG" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript> <!-- End Google Tag Manager (noscript) --> <div class="navbar navbar-inverse navbar-fixed-top" role="navigation"> <a href="https://github.com/nextflow-io/nextflow" target="_blank" class="hidden-xs"> <img style="position: absolute; top: 0; right: 0; border: 0;" src="/img/forkme_right_darkblue_121621.png" alt="Fork me on GitHub"> </a> <div class="container"> <div class="navbar-header"> <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse"> <span class="sr-only">Toggle navigation</span> <span class="icon-bar"></span> <span class="icon-bar"></span> <span class="icon-bar"></span> </button> <a class="navbar-brand" href="/index.html"> <img src="/img/nextflow.svg" title="Nextflow"> </a> </div> <div class="navbar-collapse collapse"> <ul class="nav navbar-nav"> <li class="show animated flipInX"> <a href="/docs/latest/index.html">Documentation</a> </li> <li class="dropdown show animated flipInX"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Examples <b class="caret"></b></a> <ul class="dropdown-menu"> <li><a href="/example1.html">Basic pipeline</a></li> <li><a href="/example2.html">Mixing scripting languages</a></li> <li><a href="/example3.html">BLAST pipeline</a></li> <li><a href="/example4.html">RNA-Seq pipeline</a></li> <li><a href="/example5.html">Machine Learning pipeline</a></li> <li> <a href="https://github.com/nextflow-io/rnaseq-nf" target="_blank">, Simple RNAseq pipeline, <i class="fa fa-sm fa-external-link" aria-hidden="true"></i> </a> </li> </ul> </li> <li class="show animated flipInX"> <a href="http://training.nextflow.io">Training</a> </li> <li class="dropdown show animated flipInX"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Resources <b class="caret"></b></a> <ul class="dropdown-menu"> <li><a href="/blog.html">Blog</a></li> <li><a href="/podcasts.html">Podcast</a></li> <li><a href="/blog/2023/learn-nextflow-in-2023.html">Learn Nextflow</a></li> <li> <a href="http://nextflow-io.github.io/patterns/index.html">, Implementation patterns, <i class="fa fa-sm fa-external-link" aria-hidden="true"></i> </a> </li> <li><a href="/ambassadors.html">Nextflow Ambassadors</a></li> <li> <a href="https://www.nextflow.io/slack-invite.html">, Slack community chat, <i class="fa fa-sm fa-external-link" aria-hidden="true"></i> </a> </li> <li> <a href="https://nf-co.re">, nf-core pipelines, <i class="fa fa-sm fa-external-link" aria-hidden="true"></i> </a> </li> <li><a href="/about-us.html">About Nextflow</a></li> </ul> </li> <li class="show animated flipInX"> <a href="https://community.seqera.io/c/nextflow/5" target="_blank">, Forums, <i class="fa fa-sm fa-external-link" aria-hidden="true"></i> </a> </li> <li class="show animated flipInX"> <a href="https://github.com/nextflow-io/nextflow" title="GitHub Repository"> <i class="fa fa-github hidden-xs" aria-hidden="true"></i> <span class="visible-xs">GitHub repository</span> </a> </li> </ul> </div> </div> </div>   <div class="jumbotron"> <div class="container"> <div class="row"> <div class="col-md-6 col-sm-6"> <img src="img/home-dsl2.png" alt="Nextflow code example" class="img-responsive animated fadeInDown delay1"> </div> <div class="col-md-6 col-sm-6"> <h1 class="text-color text-center animated fadeInDown delay1">Nextflow</h1> <p class="lead text-muted text-center animated fadeInDown delay2">Data-driven computational pipelines</p> <p class="text-center animated fadeInDown">, Nextflow enables scalable and reproducible scientific workflows using software containers. It allows the,             adaptation of pipelines written in the most common scripting languages., </p> <p class="text-center animated fadeInDown">, Its fluent DSL simplifies the implementation and the deployment of complex parallel and reactive workflows,             on clouds and clusters., </p> <p class="text-center"> <a href="/docs/latest/index.html" class="btn btn-color btn-xxl animated fadeInUp delay4"> Documentation </a> <a href="https://community.seqera.io/c/nextflow/5" target="_blank" class="btn btn-outline-color btn-xxl animated fadeInUp delay5">, Community forums, <i class="fa fa-sm fa-external-link" aria-hidden="true"></i> </a> </p> </div> </div> </div> </div> <div id="summit-banner-container"> <div id="summit-banner" class="animated fadeInDown"> <div> <a href="https://summit.nextflow.io/"> <img src="img/summit-2024.jpg" alt="Nextflow Summit 2024"> </a> </div> <div> <p>Join us for the latest developments and innovations from the Nextflow world!</p> <p>, With training, a hackathon and talks from pioneers in the field, the Nextflow Summits are essential events for,           anyone using Nextflow., </p> <a href="https://summit.nextflow.io/2024/boston/" class="btn btn-color btn-xxl" style="margin-top: 8px;">Register now</a> </div> </div> </div> <div class="clearfix"></div>  <div class="container"> <div class="row"> <div class="features"> <div class="feature"> <div class="feature-body"> <i class="fa fa-cog fa-3x text-color"></i> <h4>Zero config</h4> </div> <div class="feature-hover border-color"> <p><i class="fa fa-cog fa-3x text-color"></i></p> <h4>Zero config</h4> <p class="text-muted">Just download and play with it. No installation is required.</p> </div> </div> <div class="feature"> <div class="feature-body"> <i class="fa fa-bolt fa-3x text-color"></i> <h4>Polyglot</h4> </div> <div class="feature-hover border-color"> <p><i class="fa fa-bolt fa-3x text-color"></i></p> <h4>Polyglot</h4> <p class="text-muted">, Are you a Python geek or a Perl hacker? <br>, You can start fast with it., </p> </div> </div> <div class="feature"> <div class="feature-body"> <i class="fa fa-random fa-3x text-color"></i> <h4>Concurrency</h4> </div> <div class="feature-hover border-color"> <p><i class="fa fa-random fa-3x text-color"></i></p> <h4>Concurrency</h4> <p class="text-muted">Lightweight processes with message passing, no shared memory</p> </div> </div> <div class="feature"> <div class="feature-body"> <i class="fa fa-cloud fa-3x text-color"></i> <h4>Scale easily</h4> </div> <div class="feature-hover border-color"> <p><i class="fa fa-cloud fa-3x text-color"></i></p> <h4>Scale easily</h4> <p class="text-muted">Develop on your laptop, run in the grid or scale-out in the cloud with no changes.</p> </div> </div> </div> </div> </div> <div class="container" id="Features"> <div class="row"> <div class="col-md-8 col-md-offset-2 text-center"> <hr class="double-margin"> <h1>Features</h1> <p class="lead text-muted">, Nextflow is built around the idea that Linux is the <em>lingua franca</em> of data science., </p> </div> </div> <div class="row example-blocks"> <div class="col-md-6 col-sm-6 text-left"> <span class="fa-stack fa-3x pull-left"></span> <div class="block"> <h3 class="text-center text-color">Fast prototyping</h3> <p>, Nextflow allows you to write a computational pipeline by making it simpler to put together many different,             tasks., </p> <p>, You may reuse your existing scripts and tools and you don't need to learn a new language or API to start,             using it., </p> </div> </div> <div class="col-md-6 col-sm-6 text-left-xs"> <span class="fa-stack fa-3x pull-right-xs"></span> <div class="block"> <h3 class="text-center text-color">Reproducibility</h3> <p>, Nextflow supports <a href="http://docker.io" target="_blank">Docker</a> and <a href="http://singularity.lbl.gov/" target="_blank">Singularity</a> containers technology., </p> <p>, This, along with the integration of the <a href="http://github.com" target="_blank">GitHub</a> code sharing platform,,             allows you to write self-contained pipelines, manage versions and to rapidly reproduce any former configuration., </p> </div> </div> </div> <div class="row example-blocks"> <div class="col-md-6 col-sm-6 text-left"> <span class="fa-stack fa-3x pull-left"></span> <div class="block"> <h3 class="text-center text-color">Portable</h3> <p>, Nextflow provides an abstraction layer between your pipeline's logic and the execution layer, so that it can,             be executed on multiple platforms without it changing., </p> <p>, It provides out of the box executors for GridEngine, SLURM, LSF, PBS, Moab and HTCondor batch schedulers and,             for <a href="http://kubernetes.io/" target="_blank">Kubernetes</a>,, <a href="http://aws.amazon.com" target="_blank">Amazon AWS</a>,, <a href="https://cloud.google.com/compute/" target="_blank">Google Cloud</a> and, <a href="https://azure.microsoft.com/" target="_blank">Microsoft Azure</a> platforms., </p> </div> </div> <div class="col-md-6 col-sm-6 text-left-xs"> <span class="fa-stack fa-3x pull-right-xs"></span> <div class="block"> <h3 class="text-center text-color">Unified parallelism</h3> <p>, Nextflow is based on the <em>dataflow</em> programming model which greatly simplifies writing complex distributed,             pipelines., </p> <p>, Parallelisation is implicitly defined by the processes input and output declarations. The resulting,             applications are inherently parallel and can scale-up or scale-out, transparently, without having to adapt,             to a specific platform architecture., </p> </div> </div> </div> <div class="row example-blocks"> <div class="col-md-6 col-sm-6 text-left"> <span class="fa-stack fa-3x pull-left"></span> <div class="block"> <h3 class="text-center text-color">Continuous checkpoints</h3> <p>All the intermediate results produced during the pipeline execution are automatically tracked.</p> <p>, This allows you to resume its execution, from the last successfully executed step, no matter what the reason,             was for it stopping., </p> </div> </div> <div class="col-md-6 col-sm-6 text-left-xs"> <span class="fa-stack fa-3x pull-right-xs"></span> <div class="block"> <h3 class="text-center text-color">Stream oriented</h3> <p>, Nextflow extends the Unix pipes model with a fluent DSL, allowing you to handle complex stream interactions,             easily., </p> <p>, It promotes a programming approach, based on functional composition, that results in resilient and easily,             reproducible pipelines., </p> </div> </div> </div> </div> <div id="GetStarted" class="body-inverse"> <hr class="double-margin"> <div class="container"> <div class="row"> <div class="col-sm-12"> <h2 class="text-center">Getting started</h2> <p class="lead text-muted text-center">, It can be used on any POSIX compatible system (Linux, OS X, etc)., <br>Simply follow these three steps., </p> </div> </div> <div class="row"> <div class="col-sm-4"> <div class="pricing fadeInDown"> <h3 class="text-center">Check prerequisites</h3> <p class="text-muted text-center">Java 11 or later is required</p> <div class="price">1</div> <div class="pricing-text text-center text-muted">, Make sure 11 or later is installed, <br>on your computer by using the command:, <br><code>java -version </code> </div> <div style="font-size:0.8em" class="text-muted">, Note: If you are having trouble installing or upgrading Java, <br>check out our documentation <a href="https://www.nextflow.io/docs/latest/getstarted.html#requirements" target="_blank">here</a>., </div> </div> </div> <div class="col-sm-4"> <div class="pricing fadeInDown delay1"> <h3 class="text-center">Set up</h3> <p class="text-muted text-center">Dead easy to install</p> <div class="price">2</div> <div class="pricing-text text-center text-muted">, Enter this command in your terminal:, <br><code class="white-space-normal">curl -s https://get.nextflow.io | bash</code> <br>(it creates a file <code>nextflow</code> in the current dir), </div> <div style="font-size:0.8em" class="text-muted">, Note: it can also be downloaded from <a href="https://github.com/nextflow-io/nextflow/releases" target="_blank">GitHub</a>, or installed by using <a href="https://bioconda.github.io/recipes/nextflow/README.html" target="_blank">Bioconda</a> package manager., </div> </div> </div> <div class="col-sm-4"> <div class="pricing fadeInDown delay2"> <h3 class="text-center">Launch</h3> <p class="text-muted text-center">Try a simple demo</p> <div class="price">3</div> <div class="pricing-text text-center text-muted">, Run the classic <i>Hello world</i> <br>by entering the following command:, <br><code>./nextflow run hello</code> </div> <div style="font-size:0.8em" class="text-muted"> <br><br> </div> </div> </div> </div> </div> <hr class="double-margin"> </div> <div id="WhatsNext" class="container"> <div class="row"> <div class="col-sm-8 col-sm-offset-2"> <h2 class="text-center">What's next</h2> </div> <div class="col-md-4 col-md-offset-2"> <ul class="crp-blg-list"> <li> <h3> <a href="https://www.nextflow.io/docs/latest/index.html" target="_blank">Check out the documentation</a> </h3> <div class="text-muted">The Nextflow reference manual is available at the above link.</div> </li> <li> <h3><a href="example1.html">Look at the examples</a></h3> <div class="text-muted">, Some examples are provided to help you get started. More examples on their way., </div> </li> <li> <h3><a href="presentations.html">Watch the videos</a></h3> <div class="text-muted">Here you can find some Nextflow presentations and other materials.</div> </li> </ul> </div> <div class="col-md-4 col-md-offset-1"> <ul class="crp-blg-list"> <li> <h3><a href="https://www.nextflow.io/slack-invite.html" target="_blank">Confused? Ask the community</a></h3> <div class="text-muted">A Nextflow Community Chat is available on Slack. Just click the link above.</div> </li> <li> <h3><a href="https://github.com/nextflow-io/nextflow/issues" target="_blank">Report bugs</a></h3> <div class="text-muted">Bug reports help Nextflow improve, so please report any issue you may have!</div> </li> <li> <h3><a target="_blank" href="https://github.com/nextflow-io/awesome-nextflow">Applications showcase</a></h3> <div class="text-muted">, A collection of Nextflow applications developed by the Nextflow users community., </div> </li> </ul> </div> </div> </div>  <div class="footer-wrapper"> <!-- footer wrapper --> <hr> <div class="container"> <footer> <ul class="list-inline pull-right"> <li style="position: relative; top: 5px"> <a href="https://seqera.io" target="_blank" title="Developed by Seqera"> <img src="/img/seqera-logo-wide_new.png" alt="Seqera" width="136" height="28"> </a> </li> </ul> <ul class="list-inline pull-left"> <li style="position: relative; top: 5px"> <a href="https://twitter.com/nextflowio" class="twitter-follow-button" data-show-count="false">, Follow @nextflowio, </a> </li> <li> <a href="mailto:info@nextflow.io">info@nextflow.io</a> </li> </ul> </footer> </div> </div> <!-- / footer wrapper --> <!-- Bootstrap core JavaScript, ================================================== --> <!-- Placed at the end of the document so the pages load faster --> <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script> <script src="/js/bootstrap.min.js"></script> <script src="/js/custom.js"></script> <script>,   !(function (d, s, id) {,     var js,,       fjs = d.getElementsByTagName(s)[0],,       p = /^http:/.test(d.location) ? "http" : "https";,     if (!d.getElementById(id)) {,       js = d.createElement(s);,       js.id = id;,       js.src = p + "://platform.twitter.com/widgets.js";,       fjs.parentNode.insertBefore(js, fjs);,     },   })(document, "script", "twitter-wjs");, </script> </body></html>]

    at nextflow.file.http.HttpFilesTests.read a http file (HttpFilesTests.groovy:115)
bentsherman commented 1 month ago

Though I don't know why it didn't happen on the other PR you just merged...

pditommaso commented 1 month ago

Think i've understood. pathcing it

bentsherman commented 1 month ago

I should have figured, we already patched it the same way in master: https://github.com/nextflow-io/nextflow/blob/e2e608140cdde1da39df4c911f56286015538228/modules/nf-httpfs/src/test/nextflow/file/http/HttpFilesTests.groovy#L111-L115

pditommaso commented 1 month ago

Now it's failing Google logs test. @bentsherman please have a look when you have chance

bentsherman commented 1 month ago

Fixed the google test. Then I added the other commit I mentioned and something else failed. I think it's a transient issue with quay.io