Unable to find Python Requirements.txt in ADO

[dciborow]

I am trying to leverage depandabot in my Azure DevOps repo.

I keep thinking that my issue is rooted in the fact that my ADO is not defaulted to the new URL format. So our URL is "https://msorg.visualstudio.com/msproject", not "https://dev.visualstudio.com/msorg/msproject". But, my errors seem better sticking with "dev.visualstudio.com".

So far I get the following two outputs. Fetching pip dependency files for msazure/One/_git/AGAI-IndustryAI-Template Parsing dependencies information

Then big error block.

/opt/hostedtoolcache/Ruby/2.6.6/x64/lib/ruby/gems/2.6.0/gems/dependabot-common-0.111.59/lib/dependabot/shared_helpers.rb:117:in `rescue in run_helper_subprocess': Dependabot::SharedHelpers::HelperSubprocessFailed
    from /opt/hostedtoolcache/Ruby/2.6.6/x64/lib/ruby/gems/2.6.0/gems/dependabot-common-0.111.59/lib/dependabot/shared_helpers.rb:85:in `run_helper_subprocess'
    from /opt/hostedtoolcache/Ruby/2.6.6/x64/lib/ruby/gems/2.6.0/gems/dependabot-python-0.111.59/lib/dependabot/python/file_parser.rb:152:in `block in parsed_requirement_files'

/opt/hostedtoolcache/Ruby/2.6.6/x64/lib/ruby/2.6.0/json/common.rb:156:in `parse': 767: unexpected token at '' (JSON::ParserError)
    from /opt/hostedtoolcache/Ruby/2.6.6/x64/lib/ruby/2.6.0/json/common.rb:156:in `parse'
    from /opt/hostedtoolcache/Ruby/2.6.6/x64/lib/ruby/gems/2.6.0/gems/dependabot-common-0.111.59/lib/dependabot/shared_helpers.rb:109:in `run_helper_subprocess'

# This script is designed to loop through all dependencies in a GHE, GitLab or
# Azure DevOps project, creating PRs where necessary.

require "dependabot/file_fetchers"
require "dependabot/file_parsers"
require "dependabot/update_checkers"
require "dependabot/file_updaters"
require "dependabot/pull_request_creator"
require "dependabot/omnibus"
require "gitlab"

azure_hostname = ENV["AZURE_HOSTNAME"] || "dev.azure.com"

credentials = [
  {
    "type" => "git_source",
    "host" => azure_hostname,
    "username" => "x-access-token",
    "password" => ENV["SYSTEM_ACCESSTOKEN"]
  }
]

# Full name of the repo you want to create pull requests for.
repo_name = ENV["PROJECT_PATH"] || "msorg/msproject/_git/msrepo" # namespace/project

# Directory where the base dependency files are.
directory = ENV["DIRECTORY_PATH"] || "/"

# Name of the package manager you'd like to do the update for. Options are:
package_manager = ENV["PACKAGE_MANAGER"] || "pip"

source = Dependabot::Source.new(
  provider: "azure",
  hostname: azure_hostname,
  api_endpoint: "https://#{azure_hostname}/",
  repo: repo_name,
  directory: directory,
  branch: "dciborow/dependabot",
)

##############################
# Fetch the dependency files #
##############################
puts "Fetching #{package_manager} dependency files for #{repo_name}"
fetcher = Dependabot::FileFetchers.for_package_manager(package_manager).new(
  source: source,
  credentials: credentials,
)

files = fetcher.files
commit = fetcher.commit

Azure Pipeline calling update.rb

trigger:
- main

pool:
  vmImage: 'Ubuntu-16.04'

steps:
- task: UseRubyVersion@0
  inputs:
    versionSpec: '=2.6'

- task: UsePythonVersion@0
  displayName: 'Use Python 3.7'
  inputs:
    versionSpec: 3.7

- script: |
    gem install bundler
    bundle install --retry=3 --jobs=4
    curl https://pyenv.run | bash
    export PATH="$HOME/.pyenv/bin:$PATH"
    eval "$(pyenv init -)"
    eval "$(pyenv virtualenv-init -)"

    export DEPENDABOT_NATIVE_HELPERS_PATH="$(pwd)/native-helpers"
    mkdir -p $DEPENDABOT_NATIVE_HELPERS_PATH/{terraform,python,dep,go_modules,hex,composer,npm_and_yarn}
    export PATH="$PATH:$DEPENDABOT_NATIVE_HELPERS_PATH/terraform/bin:$DEPENDABOT_NATIVE_HELPERS_PATH/python/bin:$DEPENDABOT_NATIVE_HELPERS_PATH/go_modules/bin:$DEPENDABOT_NATIVE_HELPERS_PATH/dep/bin"
    export MIX_HOME="$DEPENDABOT_NATIVE_HELPERS_PATH/hex/mix"

    cp -r $(bundle show dependabot-python)/helpers $DEPENDABOT_NATIVE_HELPERS_PATH/python/helpers

    $DEPENDABOT_NATIVE_HELPERS_PATH/python/helpers/build $DEPENDABOT_NATIVE_HELPERS_PATH/python

    bundle exec ruby update.rb
  env:
    SYSTEM_ACCESSTOKEN: $(System.AccessToken)
    ob_restore_phase: true
  displayName: 'Update dependencies'

in the root of my dir is a very simple requirements.txt

absl-py

I also have a more complex one that I would ultimately like to load generated from pip-tools pip-compile process.

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --allow-unsafe --generate-hashes --output-file=./scripts/build/requirements/core-linux.txt ./scripts/build/requirements/core.in
#
absl-py==0.10.0 \
    --hash=sha256:b20f504a7871a580be5268a18fbad48af4203df5d33dbc9272426cb806245a45 \
    --hash=sha256:ea07d7d437798bffc14f39fccec3909d251a1e76e233205ded72b71c267e0178 \
    # via tensorboard, tensorflow-gpu

[jurre]

@dciborow this seems like the error is happening when trying to parse the results from the native python helpers, it seems like those may not be installed?

Also, looks like you're running a fairly old version of dependabot-core, might be worth updating that to the latest version.

It might be easiest to try and run this from within the dependabot-core docker container, that way you can ensure that the native helpers and all other dependencies required to run dependabot are present, as quite a few dependencies are needed at runtime.

Also, I know there is some effort within Microsoft to run Dependabot as a service for ADO, I've asked someone who works on that to reach out to you about that.

[dciborow]

@jurre Thanks for the internal connect, going to pull on that thread!

Can you help me with this, "the native python helpers, it seems like those may not be installed?" suggestion?

This accounts for all my ruby experience, so I am not sure what I may be missing here? Going to try and update the version regardless in case you think thats all it needs.

[jurre]

For some ecosystems, python being one, there is some code written in the native language so we can use the package manager as a library. Those helpers are being called from the ruby code as binaries, that code will print some json to stdout which we then parse.

Those native helpers are here: https://github.com/dependabot/dependabot-core/tree/main/python/helpers

And here is the code that determines where to look for those python scripts: https://github.com/dependabot/dependabot-core/blob/main/python/lib/dependabot/python/native_helpers.rb

You'll need to export that DEPENDABOT_NATIVE_HELPERS_PATH env variable with a path to those python scripts. This is all set up already in the docker container

[dciborow]

It looks like the container may be missing an update to "git".

Leading to this missing command.

My other build has git 2.29.0, which seems to have no issues.

[jurre]

I don't think you need to do a checkout of the repo, those steps seem to be run separately from what dependabot-core does? I've never used azure pipelines so I may be off here, but dependabot should be able to fetch any files it needs via the API, and in the cases where it needs a checkout of the repo (mostly for golang projects), it will do so in the FileFetcher step in the script.

[alreichf]

Hi,

Im facing a similar issue and im trying to debug line by line(im trying to bump all the pip dependencies). Ive managed to get nuget up and running but npm and pip seem to be having too many issues at the minute

the problem seems to be at 3 places

1) there needs to be a .python-config in the codebase 2) the folder for run.py does not seem to be setup properly..(i had to change code in dependabot-python-0.125.2/lib/dependabot/python/native_helpers.rb) for the path to run.py 3) pyenv needs to be installed as well..

Please if someone could help take a look. i guess running the docker image is more easier? At the minute im using omnibus. and its azuredevops repo

Thanks, Alreich

[alreichf]

seems to have gotten it to work because these python packages also need to be installed:

pip install hashin pip install pipfile pip install poetry

as they are used by these 2 python helpers:

ls ~/.rvm/gems/ruby-2.6.6/gems/dependabot-python-0.125.2/helpers/lib/ total 16 hasher.py parser.py

im wondering if this is the right way to get dependabot integrated with python.. unless im missing something obvious :/

[alreichf]

also, had to make these changes in : lib/dependabot/python/native_helpers.rb

  def self.python_helpers_dir
    File.join(native_helpers_root, "python/helpers")
  end

  def self.native_helpers_root
    default_path = File.join(__dir__, "../../../..")
    ENV.fetch("DEPENDABOT_NATIVE_HELPERS_PATH", default_path)
  end

to

  def self.python_helpers_dir
    File.join(native_helpers_root, "helpers")
  end

  def self.native_helpers_root
    default_path = File.join(__dir__, "../../..")
    ENV.fetch("DEPENDABOT_NATIVE_HELPERS_PATH", default_path)
  end

too many changes :/.. im wondering if im going down a rabbits hole :D