pex-tool / pex

A tool for generating .pex (Python EXecutable) files, lock files and venvs.
https://docs.pex-tool.org/
Apache License 2.0
2.84k stars 266 forks source link

Spurious lockfile sync errors when writing to a known pip log file #2568

Closed benjyw closed 1 month ago

benjyw commented 1 month ago

Reproduction:

$ rm -f pip.log lockfile.json
$ python -m pex.cli lock sync  --lock lockfile.json --pip-log pip.log "ansicolors==1.1.8" "cowsay==6.1"
pex: Preserving `pip download` log at pip.log
$ python -m pex.cli lock sync  --lock lockfile.json --pip-log pip.log "ansicolors==1.1.8" 
pex: Preserving `pip download` log at pip.log
Updates for lock generated by cp311-cp311-macosx_14_0_arm64:
  Deleted cowsay 6.1
Updates to lock input requirements:
  Deleted 'cowsay==6.1'
$ python -m pex.cli lock sync  --lock lockfile.json --pip-log pip.log "ansicolors==1.1.8" 
pex: Preserving `pip download` log at pip.log
Encountered 1 error updating lockfile.json:
1.) cp311-cp311-macosx_14_0_arm64: No distribution metadata found for cowsay==6.1.
Given distribution metadata for:
ansicolors==1.1.8
$ rm -f pip.log 
$ python -m pex.cli lock sync  --lock lockfile.json --pip-log pip.log "ansicolors==1.1.8" 
pex: Preserving `pip download` log at pip.log
No updates for lock generated by cp311-cp311-macosx_14_0_arm64.
$ 

This doesn't happen if --pip-log is not set.

My understanding of what is happening: when setting --pip-log pip.log pex appends to the file. So presumably pex is trying to parse lines from previous pip runs, and is confused when it sees data for requirements that aren't in the current set of input requirements.

benjyw commented 1 month ago

Is this just a matter of setting self._offset to the file length in Tailer before calling start()?

jsirois commented 1 month ago

No, its here: https://github.com/pex-tool/pex/blob/d94fa6a5051cacd1ff37bba13f40db9f9e1ffe58/pex/resolver.py#L79-L96 https://github.com/pex-tool/pex/blob/d94fa6a5051cacd1ff37bba13f40db9f9e1ffe58/pex/resolver.py#L113-L119

I handle the multi-target case correctly (say you sepcify 2 --complete-platforms), but not the single target common case.