Initial setup

[simonw]

This will use shot-scraper and GitHub Actions to take screenshots needed for the Django documentation.

It will work in a similar way to https://github.com/simonw/datasette-screenshots

[simonw]

Amusing bug... the first GitHub Actions run took this screenshot:

https://github.com/simonw/django-screenshots/blob/7c473ef5ebd137f307f464b92d3f5389b9af988b/screenshots/admin-index.png

Looks like it got a screenshot of some other piece of internal GitHub Actions infrastructure?

[simonw]

OK, so the problem here is that I want to run a localhost server inside GitHub Actions, run some requests against it and then shut it down again.

Other people seem to have solved this by running Docker containers inside GitHub Actions, so that's an option: https://stackoverflow.com/questions/60079324/running-localhost-server-for-unit-tests-executed-through-github-actions

[simonw]

This is what I'm doing now, which has the bonus that it runs locally on my laptop too:

https://github.com/simonw/django-screenshots/blob/430b722d923a89707d2d49204e8ffb144d5a928d/generate.py#L17-L38

[simonw]

https://github.com/microsoft/playwright/issues/14814#issuecomment-1154581680 reports port 3000 working.

[simonw]

I'm going to poke around in https://github.com/simonw/datasette/actions/workflows/tmate.yml and see if I can figure this out there.

[simonw]

In that test session this worked:

pip install django
django-admin startproject hello
cd hello
./manage.py migrate
./manage.py runserver 8001 &
curl localhost:8001

So running Django on port 8001 in the environment appears to be OK.

[simonw]

I am consistently getting this error now:

    raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None
playwright._impl._errors.Error: Page.goto: net::ERR_CONNECTION_REFUSED at http://localhost:8001/admin/?_user=admin
Call log:
navigating to "http://localhost:8001/admin/?_user=admin", waiting until "load"
Traceback (most recent call last):
  File "/home/runner/work/django-screenshots/django-screenshots/generate.py", line 29, in <module>
    subprocess.run([
  File "/opt/hostedtoolcache/Python/3.12.6/x64/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['shot-scraper', 'multi', 'shots.yml']' returned non-zero exit status 1.

The key bit is Error: Page.goto: net::ERR_CONNECTION_REFUSED at http://localhost:8001/admin/?_user=admin.

[simonw]

https://stackoverflow.com/a/74341376/6083 suggested this change:

https://github.com/simonw/django-screenshots/blob/cbaaa3bfd4105f0b9d0b7fef73270d637abcd0ab/.github/workflows/shots.yml#L11-L13

Adding container: ubuntu gave me this error instead:

[simonw]

I have replicated the problem on my laptop, it might not be GitHub Actions after all:

django-screenshots % python generate.py
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  No migrations to apply.
Traceback (most recent call last):
  File "/Users/simon/.local/share/virtualenvs/django-screenshots-0ofZvdSn/bin/shot-scraper", line 8, in <module>
    sys.exit(cli())
  File "/Users/simon/.local/share/virtualenvs/django-screenshots-0ofZvdSn/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/simon/.local/share/virtualenvs/django-screenshots-0ofZvdSn/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/simon/.local/share/virtualenvs/django-screenshots-0ofZvdSn/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/simon/.local/share/virtualenvs/django-screenshots-0ofZvdSn/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/simon/.local/share/virtualenvs/django-screenshots-0ofZvdSn/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/simon/.local/share/virtualenvs/django-screenshots-0ofZvdSn/lib/python3.10/site-packages/shot_scraper/cli.py", line 543, in multi
    take_shot(
  File "/Users/simon/.local/share/virtualenvs/django-screenshots-0ofZvdSn/lib/python3.10/site-packages/shot_scraper/cli.py", line 1146, in take_shot
    response = page.goto(url)
  File "/Users/simon/.local/share/virtualenvs/django-screenshots-0ofZvdSn/lib/python3.10/site-packages/playwright/sync_api/_generated.py", line 8843, in goto
    self._sync(
  File "/Users/simon/.local/share/virtualenvs/django-screenshots-0ofZvdSn/lib/python3.10/site-packages/playwright/_impl/_sync_base.py", line 115, in _sync
    return task.result()
  File "/Users/simon/.local/share/virtualenvs/django-screenshots-0ofZvdSn/lib/python3.10/site-packages/playwright/_impl/_page.py", line 524, in goto
    return await self._main_frame.goto(**locals_to_params(locals()))
  File "/Users/simon/.local/share/virtualenvs/django-screenshots-0ofZvdSn/lib/python3.10/site-packages/playwright/_impl/_frame.py", line 145, in goto
    await self._channel.send("goto", locals_to_params(locals()))
  File "/Users/simon/.local/share/virtualenvs/django-screenshots-0ofZvdSn/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 59, in send
    return await self._connection.wrap_api_call(
  File "/Users/simon/.local/share/virtualenvs/django-screenshots-0ofZvdSn/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call
    raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None
playwright._impl._errors.Error: Page.goto: net::ERR_CONNECTION_REFUSED at http://localhost:8001/admin/?_user=admin
Call log:
navigating to "http://localhost:8001/admin/?_user=admin", waiting until "load"

Traceback (most recent call last):
  File "/Users/simon/Dropbox/Development/django-screenshots/generate.py", line 32, in <module>
    subprocess.run(
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['shot-scraper', 'multi', 'shots.yml']' returned non-zero exit status 1.

[simonw]

The server was failing to start!

I did this:

# Start the Django development server
server_process = subprocess.Popen(
    ["./manage.py", "runserver", "8003"],
    cwd=testproject_dir,
    # stdout=subprocess.PIPE,
    # stderr=subprocess.PIPE
)

And now:

ModuleNotFoundError: No module named 'screenshots'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/simon/.local/share/virtualenvs/django-screenshots-0ofZvdSn/lib/python3.10/site-packages/django/utils/autoreload.py", line 64, in wrapper
    fn(*args, **kwargs)
  File "/Users/simon/.local/share/virtualenvs/django-screenshots-0ofZvdSn/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 143, in inner_run
    handler = self.get_handler(*args, **options)
  File "/Users/simon/.local/share/virtualenvs/django-screenshots-0ofZvdSn/lib/python3.10/site-packages/django/contrib/staticfiles/management/commands/runserver.py", line 31, in get_handler
    handler = super().get_handler(*args, **options)
  File "/Users/simon/.local/share/virtualenvs/django-screenshots-0ofZvdSn/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 79, in get_handler
    return get_internal_wsgi_application()
  File "/Users/simon/.local/share/virtualenvs/django-screenshots-0ofZvdSn/lib/python3.10/site-packages/django/core/servers/basehttp.py", line 50, in get_internal_wsgi_application
    raise ImproperlyConfigured(

[simonw]

It works! Shots so far are here: https://github.com/simonw/django-screenshots/tree/main/screenshots