Bizarreries dans la suite de tests

Ce n'est pas la première fois que je remarque des erreurs intermittentes durant les tests.

Je pense qu'il y a des bizarreries liées à 2 choses principalement: le côté asynchrone des tests d'une part (et probablement l'ordre aléatoire également), et aussi (hypothèse) des tests qui pourraient se connecter à des serveurs distants.

J'ouvre ce ticket pour les collecter dans le temps, et voir si un pattern clair apparaît à un moment.

Exemple concret

La PR #1705 est un exemple criant. La modification ne concerne rien dans le code, et pourtant on a une série d'échecs.

Le log CI d'origine est https://app.circleci.com/pipelines/github/etalab/transport-site/1798/workflows/418500eb-8b6d-4fae-b197-b9cab4ed6ee6/jobs/27345/parallel-runs/0/steps/0-102.

Je crée un gist pour ne pas perdre l'information au bout d'un moment:

https://gist.github.com/thbar/686f1602dd1d954599b549b66bccd0ed

En zoomé, on remarque ces points importants sur la suite finale (application apps/transport):

==> transport
Excluding tags: [:external, :pending]

..........................................

  1) test GET /dataset/:id  (TransportWeb.SeoMetadataTest)
     apps/transport/test/transport_web/controllers/seo_test.exs:85
     ** (CaseClauseError) no case clause matching: {:error, %Jason.DecodeError{data: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n", position: 0, token: nil}}
     code: title = conn |> get("/datasets/horaires-et-arrets-du-reseau-irigo-format-gtfs") |> html_response(200) |> title
     stacktrace:
       (datagouvfr 0.1.0) lib/datagouvfr/client/reuses.ex:12: Datagouvfr.Client.Reuses.get/1
       (transport 0.0.1) lib/transport_web/controllers/dataset_controller.ex:39: TransportWeb.DatasetController.details/2
       (transport 0.0.1) lib/transport_web/controllers/dataset_controller.ex:1: TransportWeb.DatasetController.action/2
       (transport 0.0.1) lib/transport_web/controllers/dataset_controller.ex:1: TransportWeb.DatasetController.phoenix_controller_pipeline/2
       (phoenix 1.5.8) lib/phoenix/router.ex:352: Phoenix.Router.__call__/2
       (transport 0.0.1) lib/transport_web/router.ex:1: TransportWeb.Router.call/2
       (transport 0.0.1) lib/plug/router.ex:284: TransportWeb.Plugs.Router.dispatch/2
       (transport 0.0.1) lib/transport_web/plugs/router.ex:1: TransportWeb.Plugs.Router.plug_builder_call/2
       (transport 0.0.1) lib/transport_web/endpoint.ex:1: TransportWeb.Endpoint.plug_builder_call/2
       (transport 0.0.1) lib/transport_web/endpoint.ex:1: TransportWeb.Endpoint."call (overridable 3)"/2
       (transport 0.0.1) lib/transport_web/endpoint.ex:1: TransportWeb.Endpoint.call/2
       (phoenix 1.5.8) lib/phoenix/test/conn_test.ex:225: Phoenix.ConnTest.dispatch/5
       test/transport_web/controllers/seo_test.exs:86: (test)

.........10:30:26.801 request_id=Fo97iZqCMy6ZWREABWYC [warn] 
A serializer was not configured for content-type 'text/html'.

To remove this warning for this content-type, add the following to your `config.exs` file:

    config :oauth2,
      serializers: %{
        "text/html" => MySerializer
      }

To remove this warning entirely, add the following to you `config.exs` file:

    config :oauth2,
      warn_missing_serializer: false

  2) test I can see my datasets (TransportWeb.ResourceControllerTest)
     apps/transport/test/transport_web/controllers/resource_controller_test.exs:24
     ** (Protocol.UndefinedError) protocol Enumerable not implemented for "<!DOCTYPE html>\n<html style=\"height:100%;\">\n<head>\n\t<meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\">\n\t<title>data.gouv.fr - La plateforme ouverte des données publiques françaises</title>\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\n    <style>\n        body {\n            font-family: Helvetica, Arial, sans-serif;\n            padding: 0;\n            margin: 0;\n        }\n        #center {\n            width: 100%;\n            text-align: center;\n            position: absolute;\n            top: 25%;\n        }\n        img {\n            width: 399px;\n            height: 210px;\n        }\n        #links {\n            width: 100%;\n            text-align: center;\n            position: absolute;\n            top: 95%;\n            font-size: 12px;\n        }\n    </style>\n\n</head>\n<body>\n    <div id=\"center\">\n        <img src=\"/img/datagouv-logo-2020.png\">\n        <p><strong>La plateforme ouverte des données publiques françaises.</strong></p>\n        <p>🤕&nbsp;<i>Site en cours de maintenance, nous faisons le maximum pour revenir en ligne rapidement.</i></p>\n    </div>\n    <div id=\"links\">\n        Vous pouvez aussi <a href=\"/game.html\">jouer à un petit jeu</a> en attendant, ou regarder le <a href=\"https://github.com/etalab/sorry.data.gouv.fr\">code source de cette page</a>.\n    </div>\n</body>\n</html>\n" of type BitString. This protocol is implemented for the following type(s): Ecto.Adapters.SQL.Stream, Postgrex.Stream, DBConnection.PrepareStream, DBConnection.Stream, Timex.Interval, Floki.HTMLTree, Scrivener.Page, Date.Range, File.Stream, Function, GenEvent.Stream, HashDict, HashSet, IO.Stream, List, Map, MapSet, Range, Stream
     code: |> get("/resources/update/datasets")
     stacktrace:
       (elixir 1.10.4) lib/enum.ex:1: Enumerable.impl_for!/1
       (elixir 1.10.4) lib/enum.ex:141: Enumerable.reduce/3
       (elixir 1.10.4) lib/enum.ex:3383: Enum.map/2
       (db 0.1.0) lib/db/dataset.ex:570: DB.Dataset.user_org_datasets/1
       (transport 0.0.1) lib/transport_web/controllers/resource_controller.ex:112: TransportWeb.ResourceController.assign_or_flash/4
       (transport 0.0.1) lib/transport_web/controllers/resource_controller.ex:43: TransportWeb.ResourceController.datasets_list/2
       (transport 0.0.1) lib/transport_web/controllers/resource_controller.ex:1: TransportWeb.ResourceController.action/2
       (transport 0.0.1) lib/transport_web/controllers/resource_controller.ex:1: TransportWeb.ResourceController.phoenix_controller_pipeline/2
       (phoenix 1.5.8) lib/phoenix/router.ex:352: Phoenix.Router.__call__/2
       (transport 0.0.1) lib/transport_web/router.ex:1: TransportWeb.Router.call/2
       (transport 0.0.1) lib/plug/router.ex:284: TransportWeb.Plugs.Router.dispatch/2
       (transport 0.0.1) lib/transport_web/plugs/router.ex:1: TransportWeb.Plugs.Router.plug_builder_call/2
       (transport 0.0.1) lib/transport_web/endpoint.ex:1: TransportWeb.Endpoint.plug_builder_call/2
       (transport 0.0.1) lib/transport_web/endpoint.ex:1: TransportWeb.Endpoint."call (overridable 3)"/2
       (transport 0.0.1) lib/transport_web/endpoint.ex:1: TransportWeb.Endpoint.call/2
       (phoenix 1.5.8) lib/phoenix/test/conn_test.ex:225: Phoenix.ConnTest.dispatch/5
       test/transport_web/controllers/resource_controller_test.exs:27: (test)

.........

  3) test I can list available datasets to find and download transport data (TransportWeb.NavTest)
     apps/transport/test/transport_web/controllers/nav_test.exs:29
     ** (CaseClauseError) no case clause matching: {:error, %Jason.DecodeError{data: "<!DOCTYPE html>\n<html style=\"height:100%;\">\n<head>\n\t<meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\">\n\t<title>data.gouv.fr - La plateforme ouverte des données publiques françaises</title>\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\n    <style>\n        body {\n            font-family: Helvetica, Arial, sans-serif;\n            padding: 0;\n            margin: 0;\n        }\n        #center {\n            width: 100%;\n            text-align: center;\n            position: absolute;\n            top: 25%;\n        }\n        img {\n            width: 399px;\n            height: 210px;\n        }\n        #links {\n            width: 100%;\n            text-align: center;\n            position: absolute;\n            top: 95%;\n            font-size: 12px;\n        }\n    </style>\n\n</head>\n<body>\n    <div id=\"center\">\n        <img src=\"/img/datagouv-logo-2020.png\">\n        <p><strong>La plateforme ouverte des données publiques françaises.</strong></p>\n        <p>🤕&nbsp;<i>Site en cours de maintenance, nous faisons le maximum pour revenir en ligne rapidement.</i></p>\n    </div>\n    <div id=\"links\">\n        Vous pouvez aussi <a href=\"/game.html\">jouer à un petit jeu</a> en attendant, ou regarder le <a href=\"https://github.com/etalab/sorry.data.gouv.fr\">code source de cette page</a>.\n    </div>\n</body>\n</html>\n", position: 0, token: nil}}
     code: conn = click_link_by_text(conn, html, "Horaires Angers")
     stacktrace:
       (datagouvfr 0.1.0) lib/datagouvfr/client/reuses.ex:12: Datagouvfr.Client.Reuses.get/1
       (transport 0.0.1) lib/transport_web/controllers/dataset_controller.ex:39: TransportWeb.DatasetController.details/2
       (transport 0.0.1) lib/transport_web/controllers/dataset_controller.ex:1: TransportWeb.DatasetController.action/2
       (transport 0.0.1) lib/transport_web/controllers/dataset_controller.ex:1: TransportWeb.DatasetController.phoenix_controller_pipeline/2
       (phoenix 1.5.8) lib/phoenix/router.ex:352: Phoenix.Router.__call__/2
       (transport 0.0.1) lib/transport_web/router.ex:1: TransportWeb.Router.call/2
       (transport 0.0.1) lib/plug/router.ex:284: TransportWeb.Plugs.Router.dispatch/2
       (transport 0.0.1) lib/transport_web/plugs/router.ex:1: TransportWeb.Plugs.Router.plug_builder_call/2
       (transport 0.0.1) lib/transport_web/endpoint.ex:1: TransportWeb.Endpoint.plug_builder_call/2
       (transport 0.0.1) lib/transport_web/endpoint.ex:1: TransportWeb.Endpoint."call (overridable 3)"/2
       (transport 0.0.1) lib/transport_web/endpoint.ex:1: TransportWeb.Endpoint.call/2
       (phoenix 1.5.8) lib/phoenix/test/conn_test.ex:225: Phoenix.ConnTest.dispatch/5
       test/transport_web/controllers/nav_test.exs:43: (test)

...............................10:30:31.225 [warn] Cannot process nil url (url) at the moment. Skipping.
......................................

Finished in 21.4 seconds
48 doctests, 100 tests, 3 failures, 16 excluded

Randomized with seed 438733

etalab / transport-site

Bizarreries dans la suite de tests #1706

Exemple concret