search

linkchecker / linkchecker

check links in web documents or full websites
https://linkchecker.github.io/linkchecker/
GNU General Public License v2.0
889 stars 147 forks source link

new PyPI release (9.4.0) #4

Closed anarcat closed 4 years ago

anarcat commented 7 years ago

so a key issue that is missing is a PyPI release.

i have pushed the 9.3.1 tag, but that's mostly cosmetic - the release hasn't been pushed to PyPI because we don't have access. i've reviewed the process documented here:

http://code.activestate.com/lists/python-distutils-sig/24089/

and it seems we can ask for access on ... ahem... sourceforge. so i've requested a password reset and will do that there. :)

anarcat commented 7 years ago

i requested access in this ticket, we'll see where it goes: https://sourceforge.net/p/pypi/support-requests/709/

considering how many requests are piled up in there, i'm not holding my breath... :/

mgedmin commented 7 years ago

Suggestion: let's register linkchecker2 on PyPI for now, temporarily, so we can get fixes out to users without delay.

If PyPI admins ever decide to give us access to the original linkchecker project, we can make linkchecker2 an empty package that depends on linkchecker.

anarcat commented 7 years ago

sounds good.

fuzzy76 commented 7 years ago

FWIW, I see these kinds of requests on their Github tracker as well.

bexelbie commented 7 years ago

While not ideal, continuing to cut releases on GH will prompt some distributions to package newer code. I realize this means that we would be in the weird situation where pypi was behind other forms of distribution, but at least we would be able to push the code out.

anarcat commented 7 years ago

note that the new Warehouse deployment on PyPI.org may have different rules that could work around the issues we have with abandoned software. see:

https://pypi.org/help/#project-name-claim

there's PEP-0541 under way to fix this, apparently.

anarcat commented 7 years ago

HEADS UP: i'll look at making a release, including on PyPI, later tomorrow or next week. thanks to @fuzzy76 I've also tried my luck at transfering PyPI ownership in https://github.com/pypa/pypi-legacy/issues/725 - if that fails, I'll just use the linkcheck namespace for now.

i've also asked the folks behind the linkchecker orga on github to see if they could transfer ownership as well, not sure how i missed that, see https://github.com/linkchecker/linkchecker.github.io/issues/2

mgoodhand commented 6 years ago

Sounds good. linkcheck would be better anyway, as it matches the module name.

anarcat commented 6 years ago

yep - and it sure looks like this is what will happen - no answer from the linkchecker orga or the pypa people yet, so I guess we're on our own for that next release... i'll give it another week.

legrostdg commented 6 years ago

@anarcat is there any update on this issue? Thanks!

anarcat commented 6 years ago

On 2017-12-12 17:49:54, legrostdg wrote:

@anarcat is there any update on this issue? Thanks!

unfortunately, i'm swamped with work and i am unlikely to do a release before the new year.

apologies for those who wanted a new release for the holidays season. :)

pomali commented 6 years ago

if there is problem with pypi namespace you could temporarily just hint people to install using pip install git+https://github.com/linkcheck/linkchecker.git in README

legrostdg commented 6 years ago

Would it be possible to at least update the Debian package to use this repo?

legrostdg commented 6 years ago

(with a new release)

anarcat commented 6 years ago

sure. if i had time. which i don't right now, as you might have noticed. :)

sorry - i can barely keep up with the PRs that are coming in here... dealing with administrative annoyances such as this is not so exciting, so it gets lower priority...

anarcat commented 6 years ago

i've created issue #136 to decide what we do about the namespace issue in general. there i propose using linkcheck everywhere, please give feedback there.

anarcat commented 6 years ago

as a stopgap measure, I have released 9.4.0 on github and debian, but still no PyPI release for now. see #141 for more information on that process.

anarcat commented 6 years ago

note that the new PyPI is in beta now, and they are explicitly talking about tackling this backlog, so there's hope. see https://lwn.net/SubscriberLink/751458/6d1a8f9a3fca1419/ for example.

anarcat commented 6 years ago

so a quick update here: it looks like there are very few claims left in the PyPI legacy issue queue (19):

https://github.com/pypa/pypi-legacy/issues

In https://github.com/pypa/warehouse/issues/3369, they discuss where to put those issues, with a broader discussion of what to do exactly with them in the future in https://github.com/pypa/warehouse/issues/1506. It will probably be associated with a ticket system or github issue queue.

It seems that processing of that queue has stalled since December, as the PyPI folks scrambled to get Warehouse launched. The only issue that was processed between our submission and now is https://github.com/pypa/pypi-legacy/issues/750 so it doesn't look like others are getting preferential treatment. I've renamed the issue to follow the ad-hoc convention and hope this will be resolved soon...

mitchellkrogza commented 6 years ago

Can this possibly be put in the main README?

pip install --user git+https://github.com/linkcheck/linkchecker.git
linkchecker https://mysite.com
thorade commented 6 years ago

@mitchellkrogza One could also use

pip install --upgrade https://github.com/linkchecker/linkchecker/archive/master.zip

Also works for those who do not have git installed and on the path.

anarcat commented 5 years ago

rah. so the repository where we made our claim has been archived: https://github.com/pypa/pypi-legacy/issues/725

i originally thought the issues there would be processed or migrated or something, but nothing of the sort has happened. it seems we need to reopen an issue in a new repository, in warehouse. there are a bunch of issues there which we could take our example from:

https://github.com/pypa/warehouse/issues?q=is%3Aissue+is%3Aopen+label%3A%22PEP+541%22

anyone wants to tackle that bit of administrative overhead?

the next step, in other words, is to take https://github.com/pypa/pypi-legacy/issues/725 and reopen it in https://github.com/pypa/warehouse, following the existing examples and PEP 541 as a ground rule.

Linkchecker qualifies as an Abandoned project and GitHub organization members should be granted access to the linkchecker project on PyPI. anarcat and mgedmin both have PyPI account that can be granted as admin to resolve this issue. Any GitHub user can file this issue and coordinate the recovery.

mgedmin commented 5 years ago

I gave it a try: https://github.com/pypa/warehouse/issues/6136

anarcat commented 5 years ago

i poked the pypa people, again. sigh...

anarcat commented 5 years ago

oh god, this unblocked! we have ownership! i logged in pypi (had to use chrome because firefox 2FA fail) and i see the project!

all of a sudden, we should make a release!!! do we want to push things out as they are now? or just push 9.4? or what do we do!

so exciting!

mgedmin commented 5 years ago

Let's twine upload an sdist built from the v9.4.0 tag and see what happens.

mgedmin commented 5 years ago

I ran check-manifest in the v9.4.0 tree and it says a bunch of files would be missing from the sdist:

``` missing from VCS: MANIFEST missing from sdist: .project .pydevproject Dockerfile dev-requirements.txt doc/contributing.mdwn doc/development.mdwn doc/dot-travis-osx.yml doc/linkchecker.1.html.diff doc/linkcheckerrc.5.html.diff doc/web doc/web/Makefile doc/web/app.yaml doc/web/content doc/web/content/donations.md doc/web/content/faq.md doc/web/content/index.md doc/web/content/other.md doc/web/content/plugins.md doc/web/hooks doc/web/hooks/__hooks__.py doc/web/media doc/web/media/css doc/web/media/css/print.css doc/web/media/css/pygment_trac.css doc/web/media/css/stylesheet.css doc/web/media/favicon.ico doc/web/media/images doc/web/media/images/back_disabled.png doc/web/media/images/back_enabled.png doc/web/media/images/back_enabled_hover.png doc/web/media/images/body-bg.jpg doc/web/media/images/download-button.png doc/web/media/images/forward_disabled.png doc/web/media/images/forward_enabled.png doc/web/media/images/forward_enabled_hover.png doc/web/media/images/github-button.png doc/web/media/images/header-bg.jpg doc/web/media/images/highlight-bg.jpg doc/web/media/images/hr.png doc/web/media/images/logo128x128.png doc/web/media/images/logo16x16.png doc/web/media/images/logo32x32.png doc/web/media/images/logo48x48.png doc/web/media/images/logo57x57.png doc/web/media/images/logo64x64.png doc/web/media/images/shot1.png doc/web/media/images/shot1_thumb.jpg doc/web/media/images/shot2.png doc/web/media/images/shot2_thumb.jpg doc/web/media/images/shot3.png doc/web/media/images/shot3_thumb.jpg doc/web/media/images/sidebar-bg.jpg doc/web/media/images/sort_asc.png doc/web/media/images/sort_asc_disabled.png doc/web/media/images/sort_both.png doc/web/media/images/sort_desc.png doc/web/media/images/sort_desc_disabled.png doc/web/media/man1 doc/web/media/man1/linkchecker.1.html doc/web/media/man1/linkchecker.1.html.rej doc/web/media/man5 doc/web/media/man5/linkcheckerrc.5.html doc/web/templates doc/web/templates/default.html doc/web/wokconfig requirements.txt robots.txt tests/checker/data/Bookmarks tests/checker/data/file.doc tests/checker/data/file.markdown tests/checker/data/file.pdf tests/checker/data/file.php tests/checker/data/file.wml tests/checker/data/plist_binary/Bookmarks.plist tests/checker/data/plist_xml/Bookmarks.plist tests/checker/data/sitemap.xml tests/checker/data/sitemapindex.xml tests/checker/data/Мошкова.bin tests/configuration/data tests/configuration/data/config0.ini tests/configuration/data/config1.ini tests/configuration/data/config2.ini tox.ini windows windows/build.bat windows/clean.bat windows/dist.bat windows/linkchecker.cer windows/linkchecker.pfx windows/linkchecker.pvk windows/makecert.bat windows/makelccert.bat windows/test.bat ```

It's probably fine. I'd maybe like to download a 9.3.0 sdist from PyPI and see what files it contains and what files it omits.

laoshaw commented 4 years ago

Installing 9.4.0 on ubuntu 18.04 did not work for me:

pip install --user git+https://github.com/linkchecker/linkchecker.git@v9.4.0
Collecting git+https://github.com/linkchecker/linkchecker.git@v9.4.0
  Cloning https://github.com/linkchecker/linkchecker.git (to revision v9.4.0) to /tmp/pip-req-build-9stjmlut
  Running command git clone -q https://github.com/linkchecker/linkchecker.git /tmp/pip-req-build-9stjmlut
  Running command git checkout -q afad6c80a2b4f3ea6f3ddf880758105fefbcf432
Requirement already satisfied: dnspython in /home/test/.local/lib/python3.6/site-packages (from LinkChecker==9.4.0) (1.16.0)
Requirement already satisfied: pyxdg in /usr/lib/python3/dist-packages (from LinkChecker==9.4.0) (0.25)
Requirement already satisfied: requests<2.15,>=2.2 in /home/test/.local/lib/python3.6/site-packages (from LinkChecker==9.4.0) (2.14.2)
Building wheels for collected packages: LinkChecker
  Building wheel for LinkChecker (setup.py) ... done
  Created wheel for LinkChecker: filename=LinkChecker-9.4.0-cp36-cp36m-linux_x86_64.whl size=407865 sha256=60f201d7ee33aea1e488fd60d7627c4b6105be456983572fdd5d11a23b85f83c
  Stored in directory: /tmp/pip-ephem-wheel-cache-qwz9czrh/wheels/a2/e3/52/a9a21270049350b19adda0c88b74a62100e30012cf3397f06b
Successfully built LinkChecker
Installing collected packages: LinkChecker
Successfully installed LinkChecker-9.4.0

linkchecker --help
Traceback (most recent call last):
  File "/home/test/.local/bin/linkchecker", line 36, in <module>
    from linkcheck.cmdline import print_version, print_usage, aggregate_url, \
  File "/home/test/.local/lib/python3.6/site-packages/linkcheck/cmdline.py", line 23, in <module>
    from . import checker, fileutil, strformat, plugins
  File "/home/test/.local/lib/python3.6/site-packages/linkcheck/checker/__init__.py", line 178, in <module>
    from . import (fileurl, unknownurl, ftpurl, httpurl, dnsurl,
  File "/home/test/.local/lib/python3.6/site-packages/linkcheck/checker/ftpurl.py", line 29, in <module>
    from . import proxysupport, httpurl, internpaturl, get_index_html
  File "/home/test/.local/lib/python3.6/site-packages/linkcheck/checker/httpurl.py", line 38, in <module>
    from ..HtmlParser import htmlsax
ImportError: /home/test/.local/lib/python3.6/site-packages/linkcheck/HtmlParser/htmlsax.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PyString_FromStringAndSize
cjmayo commented 4 years ago

Installing 9.4.0 on ubuntu 18.04 did not work for me:

9.4.0 (and current Git master) do not work on Python 3. I haven't tried a pip install but from the log provided it appears to have been installed for Python 3 which would be expected to fail.

anarcat commented 4 years ago

i'm confused here, what's our status? did we push 9.4.0 to pip yet? i think that's all that's needed to close this ticket. that it doesn't work on py3 is irrelevant.

@mgedmin did you twine upload? should i?

i opened #359 to discuss the next release

mgedmin commented 4 years ago

I never did any twine uploads.

https://pypi.org/project/LinkChecker/ doesn't have 9.4.

I would love it if you would twine upload a 9.4.0 sdist.

phaer commented 4 years ago

If I understood correctly, @anarcat could push a new release, which would be the first one since 9.3 from 2014 on pypi? Or is there anything one could help with?

Thanks a lot for your work on such a useful project! :heart:

anarcat commented 4 years ago

If I understood correctly, @anarcat could push a new release, which would be the first one since 9.3 from 2014 on pypi? Or is there anything one could help with?

correct. i'm just too swamped to coordinate that work right now, sorry. :(

anarcat commented 4 years ago

@linkchecker/developers - it seems i'm the bottleneck here, and don't have time to followup. can i grant one of you folks access? just give me a pypi username and i can add you to the group. that said, it seems @mgedmin is already an owner on the repo, so i'm not sure i'm the only one that can do this. :)

mgedmin commented 4 years ago

Yeah, I have pypi rights, but I also have burnout and a middling sense of responsibility.

I care enough that I don't want to do a bad job in making the release, but I don't care enough to invest enough effort to do a good job.

(For projects I maintain I have invested enough upfront effort that I can type make release after being woken up at 2 am with a splitting headache, and it will do all the necessary checks and tell me all the necessary things I need to do to make a good release.)

If anyone wants to be my conscience and tell me that I ought to do the following:

then I will do it.

I'm a bit worried that check-manifest complains that a bunch of files are going to be dropped from the sdist: files in doc/, tests/, windows/, a few in the project root. I'd be less worried if dist/*.tar.gz did not have other files in doc/ and tests/

I'm also slightly worried that tox shows two test failures (something about clamav and something about robots.txt).

But maybe releasing a slightly broken 9.4 sdist to pypi is less bad than the current state of affairs (totally broken 9.3 that doesn't like new requests versions due to a busted version check)?

And at least 9.4.0 is tagged and released elsewhere, if not to PyPI.

So, please tell me to do it and I'll do it. (I did all the steps except the twine upload, fwiw.)

cjmayo commented 4 years ago

FWIW Gentoo has been using the v9.4.0 tagged archive https://github.com/linkchecker/linkchecker/archive/v9.4.0.tar.gz for over a year now with no customisation (save the bash completion) and a standard disutils based install with no bugs raised.

mgedmin commented 4 years ago

Yeah, but the github tarball will contain all the files that are missing from the sdist due to lack of MANIFEST.in rules.

cjmayo commented 4 years ago

I would say you are right to be cautious. I created an sdist tarball from the tag and adapted the Gentoo ebuild, installation failed with:

FileNotFoundError: [Errno 2] No such file or directory: b'doc/development.mdwn'
cjmayo commented 4 years ago

My mistake -- that was the Gentoo ebuild selecting development.mdwn. Once I remove that Gentoo instruction it finished installation, I can't actually install it on my machine because I have got rid of all the Python 2 versions of the supporting packages.

mgedmin commented 4 years ago

I compared the files in the 9.3 tarball on PyPI with what I get in the 9.4 tarball produced from the git tag.

The 9.4 is missing

I think it's fine to release it as is.

mgedmin commented 4 years ago

I've uploaded https://pypi.org/project/LinkChecker/9.4.0/.

laoshaw commented 4 years ago

9.4.0 release is not working for me, after doing : linkchecker http://www.example.com I got:

Traceback (most recent call last):
  File "/home/userx/.local/bin/linkchecker", line 36, in <module>
    from linkcheck.cmdline import print_version, print_usage, aggregate_url, \
  File "/home/userx/.local/lib/python3.6/site-packages/linkcheck/cmdline.py", line 23, in <module>
    from . import checker, fileutil, strformat, plugins
  File "/home/userx/.local/lib/python3.6/site-packages/linkcheck/checker/__init__.py", line 178, in <module>
    from . import (fileurl, unknownurl, ftpurl, httpurl, dnsurl,
  File "/home/userx/.local/lib/python3.6/site-packages/linkcheck/checker/ftpurl.py", line 29, in <module>
    from . import proxysupport, httpurl, internpaturl, get_index_html
  File "/home/userx/.local/lib/python3.6/site-packages/linkcheck/checker/httpurl.py", line 38, in <module>
    from ..HtmlParser import htmlsax
ImportError: /home/userx/.local/lib/python3.6/site-packages/linkcheck/HtmlParser/htmlsax.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PyString_FromStringAndSize

also during installation, my requests is downgraded:

ERROR: macaroonbakery 1.3.1 has requirement requests<3.0,>=2.18.1, but you'll have requests 2.14.2 which is incompatible.
ERROR: deadlinks 0.3.0 has requirement requests>=2.22.0, but you'll have requests 2.14.2 which is incompatible.
Installing collected packages: requests, LinkChecker
  Attempting uninstall: requests
    Found existing installation: requests 2.23.0
    Uninstalling requests-2.23.0:
      Successfully uninstalled requests-2.23.0
Successfully installed LinkChecker-9.4.0 requests-2.14.2
mgedmin commented 4 years ago

9.4.0 does not support Python 3; for that you need the current git master.

mgedmin commented 4 years ago

(I am surprised and dismayed that the installation didn't fail...)

anarcat commented 4 years ago

9.4.0 release is not working for me, after doing : linkchecker http://www.example.com I got:

before anyone else jumps this issue with problems they find in 9.4.0: don't report them here! :) open a new issue and document it properly, otherwise it makes our lives needlessly hard to triage...