Notebook Integration Tests: Running down failures

[johnjosephhorton]

This will be a thread of current notebooks that fail for some reason or another.

integration/active/test_example_notebooks.py::test_notebook_execution[agentifying_responses.ipynb] FAILED

Reason: Needs this CSV file. @rbyh can you just add the data to the notebook directly or read from a public CSV on gdrive?

[johnjosephhorton]

Full list:

integration/active/test_example_notebooks.py::test_notebook_execution[adding_metadata.ipynb] PASSED                                                                            [  2%]
integration/active/test_example_notebooks.py::test_notebook_execution[agentifying_responses.ipynb] FAILED                                                                      [  4%]
integration/active/test_example_notebooks.py::test_notebook_execution[analyze_evaluations.ipynb] PASSED                                                                        [  6%]
integration/active/test_example_notebooks.py::test_notebook_execution[batching_results.ipynb] PASSED                                                                           [  8%]
integration/active/test_example_notebooks.py::test_notebook_execution[ces_agents.ipynb] FAILED                                                                                 [ 10%]
integration/active/test_example_notebooks.py::test_notebook_execution[ces_data_edsl.ipynb] PASSED                                                                              [ 12%]
integration/active/test_example_notebooks.py::test_notebook_execution[cheatsheet_scenarios.ipynb] FAILED                                                                       [ 14%]
integration/active/test_example_notebooks.py::test_notebook_execution[comparing_model_responses.ipynb] PASSED                                                                  [ 16%]
integration/active/test_example_notebooks.py::test_notebook_execution[concept_induction.ipynb] FAILED                                                                          [ 18%]
integration/active/test_example_notebooks.py::test_notebook_execution[conduct_interview.ipynb] PASSED                                                                          [ 20%]
integration/active/test_example_notebooks.py::test_notebook_execution[create_models.ipynb] PASSED                                                                              [ 22%]
integration/active/test_example_notebooks.py::test_notebook_execution[critique_questions.ipynb] PASSED                                                                         [ 24%]
integration/active/test_example_notebooks.py::test_notebook_execution[data_cleaning.ipynb] FAILED                                                                              [ 26%]
integration/active/test_example_notebooks.py::test_notebook_execution[data_labeling_agent.ipynb] FAILED                                                                        [ 28%]
integration/active/test_example_notebooks.py::test_notebook_execution[data_labeling_example.ipynb] FAILED                                                                      [ 30%]
integration/active/test_example_notebooks.py::test_notebook_execution[digital_twin.ipynb] FAILED                                                                               [ 32%]
integration/active/test_example_notebooks.py::test_notebook_execution[docs_questions.ipynb] FAILED                                                                             [ 34%]
integration/active/test_example_notebooks.py::test_notebook_execution[edsl_components.ipynb] FAILED                                                                            [ 36%]
integration/active/test_example_notebooks.py::test_notebook_execution[edsl_polling.ipynb] FAILED                                                                               [ 38%]
integration/active/test_example_notebooks.py::test_notebook_execution[edsl_with_cloud_providers.ipynb] FAILED                                                                  [ 40%]
integration/active/test_example_notebooks.py::test_notebook_execution[edsl_with_offline_inference_services.ipynb] FAILED                                                       [ 42%]
integration/active/test_example_notebooks.py::test_notebook_execution[evaluating_job_posts.ipynb] FAILED                                                                       [ 44%]
integration/active/test_example_notebooks.py::test_notebook_execution[example_agent_dynamic_traits.ipynb] PASSED                                                               [ 46%]
integration/active/test_example_notebooks.py::test_notebook_execution[exceptions_example.ipynb] FAILED                                                                         [ 48%]
integration/active/test_example_notebooks.py::test_notebook_execution[explore_llm_biases.ipynb] FAILED                                                                         [ 51%]
integration/active/test_example_notebooks.py::test_notebook_execution[explore_survey_contexts.ipynb] FAILED                                                                    [ 53%]
integration/active/test_example_notebooks.py::test_notebook_execution[export_survey_updates.ipynb] FAILED                                                                      [ 55%]
integration/active/test_example_notebooks.py::test_notebook_execution[free_responses.ipynb] PASSED                                                                             [ 57%]
integration/active/test_example_notebooks.py::test_notebook_execution[google_form_to_edsl.ipynb] FAILED                                                                        [ 59%]
integration/active/test_example_notebooks.py::test_notebook_execution[grading_experiment.ipynb] PASSED                                                                         [ 61%]
integration/active/test_example_notebooks.py::test_notebook_execution[hacker_news_hound.ipynb] PASSED                                                                          [ 63%]
integration/active/test_example_notebooks.py::test_notebook_execution[hiring_interviews.ipynb] FAILED                                                                          [ 65%]
integration/active/test_example_notebooks.py::test_notebook_execution[model_votes.ipynb] PASSED                                                                                [ 67%]
integration/active/test_example_notebooks.py::test_notebook_execution[model_walkoff.ipynb] FAILED                                                                              [ 69%]
integration/active/test_example_notebooks.py::test_notebook_execution[qualitative_research.ipynb] PASSED                                                                       [ 71%]
integration/active/test_example_notebooks.py::test_notebook_execution[question_extract_example.ipynb] PASSED                                                                   [ 73%]
integration/active/test_example_notebooks.py::test_notebook_execution[question_loop_scenarios.ipynb] FAILED                                                                    [ 75%]
integration/active/test_example_notebooks.py::test_notebook_execution[random_numbers.ipynb] PASSED                                                                             [ 77%]
integration/active/test_example_notebooks.py::test_notebook_execution[research_human_level_forecasting.ipynb] FAILED                                                           [ 79%]
integration/active/test_example_notebooks.py::test_notebook_execution[research_methods.ipynb] FAILED                                                                           [ 81%]
integration/active/test_example_notebooks.py::test_notebook_execution[research_random_silicon_sampling.ipynb] PASSED                                                           [ 83%]
integration/active/test_example_notebooks.py::test_notebook_execution[river_problem.ipynb] FAILED                                                                              [ 85%]
integration/active/test_example_notebooks.py::test_notebook_execution[scenario_from_pdf.ipynb] FAILED                                                                          [ 87%]
integration/active/test_example_notebooks.py::test_notebook_execution[scenariolist_unpivot.ipynb] FAILED                                                                       [ 89%]
integration/active/test_example_notebooks.py::test_notebook_execution[starter_tutorial.ipynb] FAILED                                                                           [ 91%]
integration/active/test_example_notebooks.py::test_notebook_execution[summarizing_transcripts.ipynb] FAILED                                                                    [ 93%]
integration/active/test_example_notebooks.py::test_notebook_execution[survey_memories.ipynb] PASSED                                                                            [ 95%]
integration/active/test_example_notebooks.py::test_notebook_execution[testing_training_data.ipynb] PASSED                                                                      [ 97%]
integration/active/test_example_notebooks.py::test_notebook_execution[writing_style.ipynb] PASSED                                                                              [100%]

[johnjosephhorton]

Notebook Name	Status	Issue
adding_metadata.ipynb	PASSED
agentifying_responses.ipynb	FAILED	Missing df = pd.read_csv("CCES22_Common_OUTPUT_vv_topost.csv", low_memory=False)
analyze_evaluations.ipynb	PASSED
batching_results.ipynb	PASSED
ces_agents.ipynb	FAILED	df = pd.read_csv("CCES22_Common_OUTPUT_vv_topost.csv", low_memory=False)
ces_data_edsl.ipynb	PASSED
cheatsheet_scenarios.ipynb	FAILED	Coop issue: survey.push(description = "Simple survey using the example scenario for a persona", visibility = "public")
comparing_model_responses.ipynb	PASSED
concept_induction.ipynb	FAILED	Error in filter. Exception:Function 'int' not defined, for expression 'int(score) > 0'.."- we probably just need to pass 'int' as a function for the evaluator to use, cc: @zer0dss
conduct_interview.ipynb	PASSED
create_models.ipynb	PASSED
critique_questions.ipynb	PASSED
data_cleaning.ipynb	FAILED	Coop issue
data_labeling_agent.ipynb	FAILED	Missing data: scenarios = ScenarioList.from_csv("job_posts.csv")
data_labeling_example.ipynb	FAILED
digital_twin.ipynb	FAILED
docs_questions.ipynb	FAILED
edsl_components.ipynb	FAILED
edsl_polling.ipynb	FAILED
edsl_with_cloud_providers.ipynb	FAILED
edsl_with_offline_inference_services.ipynb	FAILED
evaluating_job_posts.ipynb	FAILED
example_agent_dynamic_traits.ipynb	PASSED
exceptions_example.ipynb	FAILED
explore_llm_biases.ipynb	FAILED
explore_survey_contexts.ipynb	FAILED
export_survey_updates.ipynb	FAILED
free_responses.ipynb	PASSED
google_form_to_edsl.ipynb	FAILED
grading_experiment.ipynb	PASSED
hacker_news_hound.ipynb	PASSED
hiring_interviews.ipynb	FAILED
model_votes.ipynb	PASSED
model_walkoff.ipynb	FAILED
qualitative_research.ipynb	PASSED
question_extract_example.ipynb	PASSED
question_loop_scenarios.ipynb	FAILED
random_numbers.ipynb	PASSED
research_human_level_forecasting.ipynb	FAILED
research_methods.ipynb	FAILED
research_random_silicon_sampling.ipynb	PASSED
river_problem.ipynb	FAILED
scenario_from_pdf.ipynb	FAILED
scenariolist_unpivot.ipynb	FAILED
starter_tutorial.ipynb	FAILED
summarizing_transcripts.ipynb	FAILED
survey_memories.ipynb	PASSED
testing_training_data.ipynb	PASSED
writing_style.ipynb	PASSED

[johnjosephhorton]

@rbyh I think a lot of these are probably coop pushes that the notebook cannot make in a testing env w/ an API key. Rather than remove or wrap in an API Key, we should maybe just add a "fail_silently = True" optional argument to push?

Another class is just missing data e.g. a CSV file.

[johnjosephhorton]

If you add a "skip-execution" tag to a cell, it will be skipped when tests run cc: @rbyh :

NB - to run a single notebook test:

make test-notebooks notebook='docs/notebooks/data_cleaning.ipynb'

[rbyh]

Let's work off this sheet for fixing the notebooks: https://docs.google.com/spreadsheets/d/1Df8_d8nZGFfk5YKF6XxpM9D0GHByJLG-_BkoZcy-bPk/edit?usp=sharing

[johnjosephhorton]

@rbyh As an interim fix for CSV files in the notebooks, I have this FileStore class that's a subclass of Scenario (and hence can work with coop). This "works" (or will rather once I submit a PR that fixes a few things):

from edsl import ScenarioList, Scenario
from edsl.scenarios.FileStore import CSVFileStore

create_file_store = True

key = "3d65a9dd-04b2-4385-9fda-bc61a297d576"

if create_file_store:
    # create an example CSV file 
    s = ScenarioList.example()
    s.to_csv("example.csv")

    with open("example.csv", "r") as f:
        fs = CSVFileStore(f.name)

    # push it to coop
    info = fs.push()

# Pull it back & then load it into a ScenarioList object using existing to_tempfile method
fs = CSVFileStore.pull(key)
s = ScenarioList.from_csv(fs.to_tempfile())

@zer0dss / @apostolosfilippas - this is kind of a hack that takes advantage of the fact that Scenario object is very general and we can ecode files as strings. FWIW, I do think creating a way for users to store their data on coop has some nice advantages and it's already built. We could make this slightly more user friendly by allowing that from_csv method to also work with coop UUIDs / filestore objects. Thoughts?

[rbyh]

Getting this error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[1], line 21
     17     info = fs.push()
     20 # Pull it back & then load it into a ScenarioList object using existing to_tempfile method
---> 21 fs = CSVFileStore.pull(key)
     22 s = ScenarioList.from_csv(fs.to_tempfile())

File [~/edsl/edsl/scenarios/FileStore.py:124](http://localhost:8888/lab/tree/~/edsl/edsl/scenarios/FileStore.py#line=123), in FileStore.pull(cls, uuid)
    122 @classmethod
    123 def pull(cls, uuid):
--> 124     scenario_version = Scenario.pull(uuid)
    125     return cls.from_dict(scenario_version.to_dict())

File [~/edsl/edsl/Base.py:65](http://localhost:8888/lab/tree/~/edsl/edsl/Base.py#line=64), in PersistenceMixin.pull(cls, uuid, url)
     63 object_type = ObjectRegistry.get_object_type_by_edsl_class(cls)
     64 coop = Coop()
---> 65 return coop.get(uuid, url, object_type)

File [~/edsl/edsl/coop/coop.py:190](http://localhost:8888/lab/tree/~/edsl/edsl/coop/coop.py#line=189), in Coop.get(self, uuid, url, expected_object_type)
    178 """
    179 Retrieve an EDSL object by its uuid or its url.
    180 - If the object's visibility is private, the user must be the owner.
   (...)
    187 :return: the object instance.
    188 """
    189 uuid = self._resolve_uuid(uuid, url)
--> 190 response = self._send_server_request(
    191     uri=f"api[/v0/object](http://localhost:8888/v0/object)",
    192     method="GET",
    193     params={"uuid": uuid},
    194 )
    195 self._resolve_server_response(response)
    196 json_string = response.json().get("json_string")

File [~/edsl/edsl/coop/coop.py:62](http://localhost:8888/lab/tree/~/edsl/edsl/coop/coop.py#line=61), in Coop._send_server_request(self, uri, method, payload, params, timeout)
     58 """
     59 Send a request to the server and return the response.
     60 """
     61 url = f"{self.url}[/](http://localhost:8888/){uri}"
---> 62 timeout = max(5, (len(payload["json_string"]) // (1024 * 1024)))
     63 try:
     64     method = method.upper()

TypeError: 'NoneType' object is not subscriptable

[zer0dss]

@rbyh I'm updating edsl to fix the above error.

[rbyh]

Error is gone, thanks

[rbyh]

@johnjosephhorton But I am still getting a Coop connection error when running tests for the notebook -- I cannot skip execution of the cells at issue:

https://www.expectedparrot.com/content/b145af25-e864-40fe-b7af-a61942a16ca9

[rbyh]

(edsl-py3.11) a16174@MacBook-Air edsl % make test-notebooks notebook='filestore_example.ipynb'      
Testing notebook: filestore_example.ipynb
======================================================= test session starts =======================================================
platform darwin -- Python 3.11.7, pytest-7.4.4, pluggy-1.5.0 -- /Users/a16174/edsl/.venv/bin/python
cachedir: .pytest_cache
metadata: {'Python': '3.11.7', 'Platform': 'macOS-14.6.1-arm64-arm-64bit', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'asyncio': '0.23.8', 'html': '4.1.1', 'env': '1.1.3', 'metadata': '3.1.1', 'anyio': '4.4.0', 'Faker': '28.4.1', 'profiling': '1.7.0', 'mock': '3.14.0', 'xdist': '3.6.1'}}
rootdir: /Users/a16174/edsl/integration
configfile: pytest.ini
plugins: asyncio-0.23.8, html-4.1.1, env-1.1.3, metadata-3.1.1, anyio-4.4.0, Faker-28.4.1, profiling-1.7.0, mock-3.14.0, xdist-3.6.1
asyncio: mode=Mode.STRICT
collected 121 items / 120 deselected / 1 selected                                                                                 

integration/active/test_notebooks.py::test_notebook_execution[docs/notebooks/filestore_example.ipynb] FAILED                [100%]

============================================================ FAILURES =============================================================
_________________________________ test_notebook_execution[docs/notebooks/filestore_example.ipynb] _________________________________

notebook_path = 'docs/notebooks/filestore_example.ipynb'

    def execute_notebook(notebook_path):
        """
        Execute a Jupyter notebook and either returns True if successful or raises an exception.
        Skips cells tagged with 'skip-execution'.
        """
        with open(notebook_path) as f:
            nb = nbformat.read(f, as_version=4)
            ep = SkipTaggedCells(timeout=600, kernel_name="python3")

            try:
                # Attempt to execute the notebook
>               ep.preprocess(nb, {"metadata": {"path": os.path.dirname(notebook_path)}})

integration/active/test_notebooks.py:27: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.venv/lib/python3.11/site-packages/nbconvert/preprocessors/execute.py:103: in preprocess
    self.preprocess_cell(cell, resources, index)
integration/active/test_notebooks.py:13: in preprocess_cell
    return super().preprocess_cell(cell, resources, cell_index)
.venv/lib/python3.11/site-packages/nbconvert/preprocessors/execute.py:124: in preprocess_cell
    cell = self.execute_cell(cell, index, store_history=True)
.venv/lib/python3.11/site-packages/jupyter_core/utils/__init__.py:165: in wrapped
    return loop.run_until_complete(inner)
../.pyenv/versions/3.11.7/lib/python3.11/asyncio/base_events.py:653: in run_until_complete
    return future.result()
.venv/lib/python3.11/site-packages/nbclient/client.py:1062: in async_execute_cell
    await self._check_raise_for_error(cell, cell_index, exec_reply)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <test_notebooks.SkipTaggedCells object at 0x1042f7e10>
cell = {'cell_type': 'code', 'execution_count': 2, 'id': 'b4b3351b-d6fd-4d88-992b-a6799ac0a493', 'metadata': {'editable': Tru...b[0m: Could not connect to the server.']}], 'source': 'fs = CSVFileStore.pull("429f0b54-58da-4df0-a1bb-f6c0223ba9db")'}
cell_index = 1
exec_reply = {'buffers': [], 'content': {'ename': 'ConnectionError', 'engine_info': {'engine_id': -1, 'engine_uuid': 'eddd5507-de48...e, 'engine': 'eddd5507-de48-44a5-a444-9b30d4e7d8cb', 'started': '2024-09-12T20:42:33.626700Z', 'status': 'error'}, ...}

    async def _check_raise_for_error(
        self, cell: NotebookNode, cell_index: int, exec_reply: dict[str, t.Any] | None
    ) -> None:
        if exec_reply is None:
            return None

        exec_reply_content = exec_reply["content"]
        if exec_reply_content["status"] != "error":
            return None

        cell_allows_errors = (not self.force_raise_errors) and (
            self.allow_errors
            or exec_reply_content.get("ename") in self.allow_error_names
            or "raises-exception" in cell.metadata.get("tags", [])
        )
        await run_hook(
            self.on_cell_error, cell=cell, cell_index=cell_index, execute_reply=exec_reply
        )
        if not cell_allows_errors:
>           raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)
E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
E           ------------------
E           fs = CSVFileStore.pull("429f0b54-58da-4df0-a1bb-f6c0223ba9db")
E           ------------------
E           
E           
E           ---------------------------------------------------------------------------
E           ConnectionRefusedError                    Traceback (most recent call last)
E           File ~/edsl/.venv/lib/python3.11/site-packages/urllib3/connection.py:196, in HTTPConnection._new_conn(self)
E               195 try:
E           --> 196     sock = connection.create_connection(
E               197         (self._dns_host, self.port),
E               198         self.timeout,
E               199         source_address=self.source_address,
E               200         socket_options=self.socket_options,
E               201     )
E               202 except socket.gaierror as e:
E           
E           File ~/edsl/.venv/lib/python3.11/site-packages/urllib3/util/connection.py:85, in create_connection(address, timeout, source_address, socket_options)
E                84 try:
E           ---> 85     raise err
E                86 finally:
E                87     # Break explicitly a reference cycle
E           
E           File ~/edsl/.venv/lib/python3.11/site-packages/urllib3/util/connection.py:73, in create_connection(address, timeout, source_address, socket_options)
E                72     sock.bind(source_address)
E           ---> 73 sock.connect(sa)
E                74 # Break explicitly a reference cycle
E           
E           ConnectionRefusedError: [Errno 61] Connection refused
E           
E           The above exception was the direct cause of the following exception:
E           
E           NewConnectionError                        Traceback (most recent call last)
E           File ~/edsl/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py:789, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
E               788 # Make the request on the HTTPConnection object
E           --> 789 response = self._make_request(
E               790     conn,
E               791     method,
E               792     url,
E               793     timeout=timeout_obj,
E               794     body=body,
E               795     headers=headers,
E               796     chunked=chunked,
E               797     retries=retries,
E               798     response_conn=response_conn,
E               799     preload_content=preload_content,
E               800     decode_content=decode_content,
E               801     **response_kw,
E               802 )
E               804 # Everything went great!
E           
E           File ~/edsl/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py:495, in HTTPConnectionPool._make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
E               494 try:
E           --> 495     conn.request(
E               496         method,
E               497         url,
E               498         body=body,
E               499         headers=headers,
E               500         chunked=chunked,
E               501         preload_content=preload_content,
E               502         decode_content=decode_content,
E               503         enforce_content_length=enforce_content_length,
E               504     )
E               506 # We are swallowing BrokenPipeError (errno.EPIPE) since the server is
E               507 # legitimately able to close the connection after sending a valid response.
E               508 # With this behaviour, the received response is still readable.
E           
E           File ~/edsl/.venv/lib/python3.11/site-packages/urllib3/connection.py:398, in HTTPConnection.request(self, method, url, body, headers, chunked, preload_content, decode_content, enforce_content_length)
E               397     self.putheader(header, value)
E           --> 398 self.endheaders()
E               400 # If we're given a body we start sending that in chunks.
E           
E           File ~/.pyenv/versions/3.11.7/lib/python3.11/http/client.py:1289, in HTTPConnection.endheaders(self, message_body, encode_chunked)
E              1288     raise CannotSendHeader()
E           -> 1289 self._send_output(message_body, encode_chunked=encode_chunked)
E           
E           File ~/.pyenv/versions/3.11.7/lib/python3.11/http/client.py:1048, in HTTPConnection._send_output(self, message_body, encode_chunked)
E              1047 del self._buffer[:]
E           -> 1048 self.send(msg)
E              1050 if message_body is not None:
E              1051 
E              1052     # create a consistent interface to message_body
E           
E           File ~/.pyenv/versions/3.11.7/lib/python3.11/http/client.py:986, in HTTPConnection.send(self, data)
E               985 if self.auto_open:
E           --> 986     self.connect()
E               987 else:
E           
E           File ~/edsl/.venv/lib/python3.11/site-packages/urllib3/connection.py:236, in HTTPConnection.connect(self)
E               235 def connect(self) -> None:
E           --> 236     self.sock = self._new_conn()
E               237     if self._tunnel_host:
E               238         # If we're tunneling it means we're connected to our proxy.
E           
E           File ~/edsl/.venv/lib/python3.11/site-packages/urllib3/connection.py:211, in HTTPConnection._new_conn(self)
E               210 except OSError as e:
E           --> 211     raise NewConnectionError(
E               212         self, f"Failed to establish a new connection: {e}"
E               213     ) from e
E               215 # Audit hooks are only available in Python 3.8+
E           
E           NewConnectionError: <urllib3.connection.HTTPConnection object at 0x120245790>: Failed to establish a new connection: [Errno 61] Connection refused
E           
E           The above exception was the direct cause of the following exception:
E           
E           MaxRetryError                             Traceback (most recent call last)
E           File ~/edsl/.venv/lib/python3.11/site-packages/requests/adapters.py:667, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
E               666 try:
E           --> 667     resp = conn.urlopen(
E               668         method=request.method,
E               669         url=url,
E               670         body=request.body,
E               671         headers=request.headers,
E               672         redirect=False,
E               673         assert_same_host=False,
E               674         preload_content=False,
E               675         decode_content=False,
E               676         retries=self.max_retries,
E               677         timeout=timeout,
E               678         chunked=chunked,
E               679     )
E               681 except (ProtocolError, OSError) as err:
E           
E           File ~/edsl/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py:843, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
E               841     new_e = ProtocolError("Connection aborted.", new_e)
E           --> 843 retries = retries.increment(
E               844     method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
E               845 )
E               846 retries.sleep()
E           
E           File ~/edsl/.venv/lib/python3.11/site-packages/urllib3/util/retry.py:519, in Retry.increment(self, method, url, response, error, _pool, _stacktrace)
E               518     reason = error or ResponseError(cause)
E           --> 519     raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
E               521 log.debug("Incremented Retry for (url='%s'): %r", url, new_retry)
E           
E           MaxRetryError: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /api/v0/object?uuid=429f0b54-58da-4df0-a1bb-f6c0223ba9db (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x120245790>: Failed to establish a new connection: [Errno 61] Connection refused'))
E           
E           During handling of the above exception, another exception occurred:
E           
E           ConnectionError                           Traceback (most recent call last)
E           File ~/edsl/edsl/coop/coop.py:69, in Coop._send_server_request(self, uri, method, payload, params, timeout)
E                68 if method in ["GET", "DELETE"]:
E           ---> 69     response = requests.request(
E                70         method, url, params=params, headers=self.headers, timeout=timeout
E                71     )
E                72 elif method in ["POST", "PATCH"]:
E           
E           File ~/edsl/.venv/lib/python3.11/site-packages/requests/api.py:59, in request(method, url, **kwargs)
E                58 with sessions.Session() as session:
E           ---> 59     return session.request(method=method, url=url, **kwargs)
E           
E           File ~/edsl/.venv/lib/python3.11/site-packages/requests/sessions.py:589, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
E               588 send_kwargs.update(settings)
E           --> 589 resp = self.send(prep, **send_kwargs)
E               591 return resp
E           
E           File ~/edsl/.venv/lib/python3.11/site-packages/requests/sessions.py:703, in Session.send(self, request, **kwargs)
E               702 # Send the request
E           --> 703 r = adapter.send(request, **kwargs)
E               705 # Total elapsed time of the request (approximately)
E           
E           File ~/edsl/.venv/lib/python3.11/site-packages/requests/adapters.py:700, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
E               698         raise SSLError(e, request=request)
E           --> 700     raise ConnectionError(e, request=request)
E               702 except ClosedPoolError as e:
E           
E           ConnectionError: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /api/v0/object?uuid=429f0b54-58da-4df0-a1bb-f6c0223ba9db (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x120245790>: Failed to establish a new connection: [Errno 61] Connection refused'))
E           
E           During handling of the above exception, another exception occurred:
E           
E           ConnectionError                           Traceback (most recent call last)
E           Cell In[2], line 1
E           ----> 1 fs = CSVFileStore.pull("429f0b54-58da-4df0-a1bb-f6c0223ba9db")
E           
E           File ~/edsl/edsl/scenarios/FileStore.py:124, in FileStore.pull(cls, uuid)
E               122 @classmethod
E               123 def pull(cls, uuid):
E           --> 124     scenario_version = Scenario.pull(uuid)
E               125     return cls.from_dict(scenario_version.to_dict())
E           
E           File ~/edsl/edsl/Base.py:65, in PersistenceMixin.pull(cls, uuid, url)
E                63 object_type = ObjectRegistry.get_object_type_by_edsl_class(cls)
E                64 coop = Coop()
E           ---> 65 return coop.get(uuid, url, object_type)
E           
E           File ~/edsl/edsl/coop/coop.py:193, in Coop.get(self, uuid, url, expected_object_type)
E               181 """
E               182 Retrieve an EDSL object by its uuid or its url.
E               183 - If the object's visibility is private, the user must be the owner.
E              (...)
E               190 :return: the object instance.
E               191 """
E               192 uuid = self._resolve_uuid(uuid, url)
E           --> 193 response = self._send_server_request(
E               194     uri=f"api/v0/object",
E               195     method="GET",
E               196     params={"uuid": uuid},
E               197 )
E               198 self._resolve_server_response(response)
E               199 json_string = response.json().get("json_string")
E           
E           File ~/edsl/edsl/coop/coop.py:84, in Coop._send_server_request(self, uri, method, payload, params, timeout)
E                82         raise Exception(f"Invalid {method=}.")
E                83 except requests.ConnectionError:
E           ---> 84     raise requests.ConnectionError("Could not connect to the server.")
E                86 return response
E           
E           ConnectionError: Could not connect to the server.

.venv/lib/python3.11/site-packages/nbclient/client.py:918: CellExecutionError

During handling of the above exception, another exception occurred:

notebook_path = 'docs/notebooks/filestore_example.ipynb'

    @pytest.mark.parametrize("notebook_path", get_notebooks())
    def test_notebook_execution(notebook_path):
        """
        Test function that executes each Jupyter notebook and checks for exceptions.
        """
        print(f"Executing {notebook_path}...")
>       execute_notebook(notebook_path)

integration/active/test_notebooks.py:51: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

notebook_path = 'docs/notebooks/filestore_example.ipynb'

    def execute_notebook(notebook_path):
        """
        Execute a Jupyter notebook and either returns True if successful or raises an exception.
        Skips cells tagged with 'skip-execution'.
        """
        with open(notebook_path) as f:
            nb = nbformat.read(f, as_version=4)
            ep = SkipTaggedCells(timeout=600, kernel_name="python3")

            try:
                # Attempt to execute the notebook
                ep.preprocess(nb, {"metadata": {"path": os.path.dirname(notebook_path)}})
            except CellExecutionError as cell_error:
>               raise AssertionError(f"Execution error in {notebook_path}: {cell_error}")
E               AssertionError: Execution error in docs/notebooks/filestore_example.ipynb: An error occurred while executing the following cell:
E               ------------------
E               fs = CSVFileStore.pull("429f0b54-58da-4df0-a1bb-f6c0223ba9db")
E               ------------------
E               
E               
E               ---------------------------------------------------------------------------
E               ConnectionRefusedError                    Traceback (most recent call last)
E               File ~/edsl/.venv/lib/python3.11/site-packages/urllib3/connection.py:196, in HTTPConnection._new_conn(self)
E                   195 try:
E               --> 196     sock = connection.create_connection(
E                   197         (self._dns_host, self.port),
E                   198         self.timeout,
E                   199         source_address=self.source_address,
E                   200         socket_options=self.socket_options,
E                   201     )
E                   202 except socket.gaierror as e:
E               
E               File ~/edsl/.venv/lib/python3.11/site-packages/urllib3/util/connection.py:85, in create_connection(address, timeout, source_address, socket_options)
E                    84 try:
E               ---> 85     raise err
E                    86 finally:
E                    87     # Break explicitly a reference cycle
E               
E               File ~/edsl/.venv/lib/python3.11/site-packages/urllib3/util/connection.py:73, in create_connection(address, timeout, source_address, socket_options)
E                    72     sock.bind(source_address)
E               ---> 73 sock.connect(sa)
E                    74 # Break explicitly a reference cycle
E               
E               ConnectionRefusedError: [Errno 61] Connection refused
E               
E               The above exception was the direct cause of the following exception:
E               
E               NewConnectionError                        Traceback (most recent call last)
E               File ~/edsl/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py:789, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
E                   788 # Make the request on the HTTPConnection object
E               --> 789 response = self._make_request(
E                   790     conn,
E                   791     method,
E                   792     url,
E                   793     timeout=timeout_obj,
E                   794     body=body,
E                   795     headers=headers,
E                   796     chunked=chunked,
E                   797     retries=retries,
E                   798     response_conn=response_conn,
E                   799     preload_content=preload_content,
E                   800     decode_content=decode_content,
E                   801     **response_kw,
E                   802 )
E                   804 # Everything went great!
E               
E               File ~/edsl/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py:495, in HTTPConnectionPool._make_request(self, conn, method, url, body, headers, retries, timeout, chunked, response_conn, preload_content, decode_content, enforce_content_length)
E                   494 try:
E               --> 495     conn.request(
E                   496         method,
E                   497         url,
E                   498         body=body,
E                   499         headers=headers,
E                   500         chunked=chunked,
E                   501         preload_content=preload_content,
E                   502         decode_content=decode_content,
E                   503         enforce_content_length=enforce_content_length,
E                   504     )
E                   506 # We are swallowing BrokenPipeError (errno.EPIPE) since the server is
E                   507 # legitimately able to close the connection after sending a valid response.
E                   508 # With this behaviour, the received response is still readable.
E               
E               File ~/edsl/.venv/lib/python3.11/site-packages/urllib3/connection.py:398, in HTTPConnection.request(self, method, url, body, headers, chunked, preload_content, decode_content, enforce_content_length)
E                   397     self.putheader(header, value)
E               --> 398 self.endheaders()
E                   400 # If we're given a body we start sending that in chunks.
E               
E               File ~/.pyenv/versions/3.11.7/lib/python3.11/http/client.py:1289, in HTTPConnection.endheaders(self, message_body, encode_chunked)
E                  1288     raise CannotSendHeader()
E               -> 1289 self._send_output(message_body, encode_chunked=encode_chunked)
E               
E               File ~/.pyenv/versions/3.11.7/lib/python3.11/http/client.py:1048, in HTTPConnection._send_output(self, message_body, encode_chunked)
E                  1047 del self._buffer[:]
E               -> 1048 self.send(msg)
E                  1050 if message_body is not None:
E                  1051 
E                  1052     # create a consistent interface to message_body
E               
E               File ~/.pyenv/versions/3.11.7/lib/python3.11/http/client.py:986, in HTTPConnection.send(self, data)
E                   985 if self.auto_open:
E               --> 986     self.connect()
E                   987 else:
E               
E               File ~/edsl/.venv/lib/python3.11/site-packages/urllib3/connection.py:236, in HTTPConnection.connect(self)
E                   235 def connect(self) -> None:
E               --> 236     self.sock = self._new_conn()
E                   237     if self._tunnel_host:
E                   238         # If we're tunneling it means we're connected to our proxy.
E               
E               File ~/edsl/.venv/lib/python3.11/site-packages/urllib3/connection.py:211, in HTTPConnection._new_conn(self)
E                   210 except OSError as e:
E               --> 211     raise NewConnectionError(
E                   212         self, f"Failed to establish a new connection: {e}"
E                   213     ) from e
E                   215 # Audit hooks are only available in Python 3.8+
E               
E               NewConnectionError: <urllib3.connection.HTTPConnection object at 0x120245790>: Failed to establish a new connection: [Errno 61] Connection refused
E               
E               The above exception was the direct cause of the following exception:
E               
E               MaxRetryError                             Traceback (most recent call last)
E               File ~/edsl/.venv/lib/python3.11/site-packages/requests/adapters.py:667, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
E                   666 try:
E               --> 667     resp = conn.urlopen(
E                   668         method=request.method,
E                   669         url=url,
E                   670         body=request.body,
E                   671         headers=request.headers,
E                   672         redirect=False,
E                   673         assert_same_host=False,
E                   674         preload_content=False,
E                   675         decode_content=False,
E                   676         retries=self.max_retries,
E                   677         timeout=timeout,
E                   678         chunked=chunked,
E                   679     )
E                   681 except (ProtocolError, OSError) as err:
E               
E               File ~/edsl/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py:843, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
E                   841     new_e = ProtocolError("Connection aborted.", new_e)
E               --> 843 retries = retries.increment(
E                   844     method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
E                   845 )
E                   846 retries.sleep()
E               
E               File ~/edsl/.venv/lib/python3.11/site-packages/urllib3/util/retry.py:519, in Retry.increment(self, method, url, response, error, _pool, _stacktrace)
E                   518     reason = error or ResponseError(cause)
E               --> 519     raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
E                   521 log.debug("Incremented Retry for (url='%s'): %r", url, new_retry)
E               
E               MaxRetryError: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /api/v0/object?uuid=429f0b54-58da-4df0-a1bb-f6c0223ba9db (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x120245790>: Failed to establish a new connection: [Errno 61] Connection refused'))
E               
E               During handling of the above exception, another exception occurred:
E               
E               ConnectionError                           Traceback (most recent call last)
E               File ~/edsl/edsl/coop/coop.py:69, in Coop._send_server_request(self, uri, method, payload, params, timeout)
E                    68 if method in ["GET", "DELETE"]:
E               ---> 69     response = requests.request(
E                    70         method, url, params=params, headers=self.headers, timeout=timeout
E                    71     )
E                    72 elif method in ["POST", "PATCH"]:
E               
E               File ~/edsl/.venv/lib/python3.11/site-packages/requests/api.py:59, in request(method, url, **kwargs)
E                    58 with sessions.Session() as session:
E               ---> 59     return session.request(method=method, url=url, **kwargs)
E               
E               File ~/edsl/.venv/lib/python3.11/site-packages/requests/sessions.py:589, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
E                   588 send_kwargs.update(settings)
E               --> 589 resp = self.send(prep, **send_kwargs)
E                   591 return resp
E               
E               File ~/edsl/.venv/lib/python3.11/site-packages/requests/sessions.py:703, in Session.send(self, request, **kwargs)
E                   702 # Send the request
E               --> 703 r = adapter.send(request, **kwargs)
E                   705 # Total elapsed time of the request (approximately)
E               
E               File ~/edsl/.venv/lib/python3.11/site-packages/requests/adapters.py:700, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
E                   698         raise SSLError(e, request=request)
E               --> 700     raise ConnectionError(e, request=request)
E                   702 except ClosedPoolError as e:
E               
E               ConnectionError: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /api/v0/object?uuid=429f0b54-58da-4df0-a1bb-f6c0223ba9db (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x120245790>: Failed to establish a new connection: [Errno 61] Connection refused'))
E               
E               During handling of the above exception, another exception occurred:
E               
E               ConnectionError                           Traceback (most recent call last)
E               Cell In[2], line 1
E               ----> 1 fs = CSVFileStore.pull("429f0b54-58da-4df0-a1bb-f6c0223ba9db")
E               
E               File ~/edsl/edsl/scenarios/FileStore.py:124, in FileStore.pull(cls, uuid)
E                   122 @classmethod
E                   123 def pull(cls, uuid):
E               --> 124     scenario_version = Scenario.pull(uuid)
E                   125     return cls.from_dict(scenario_version.to_dict())
E               
E               File ~/edsl/edsl/Base.py:65, in PersistenceMixin.pull(cls, uuid, url)
E                    63 object_type = ObjectRegistry.get_object_type_by_edsl_class(cls)
E                    64 coop = Coop()
E               ---> 65 return coop.get(uuid, url, object_type)
E               
E               File ~/edsl/edsl/coop/coop.py:193, in Coop.get(self, uuid, url, expected_object_type)
E                   181 """
E                   182 Retrieve an EDSL object by its uuid or its url.
E                   183 - If the object's visibility is private, the user must be the owner.
E                  (...)
E                   190 :return: the object instance.
E                   191 """
E                   192 uuid = self._resolve_uuid(uuid, url)
E               --> 193 response = self._send_server_request(
E                   194     uri=f"api/v0/object",
E                   195     method="GET",
E                   196     params={"uuid": uuid},
E                   197 )
E                   198 self._resolve_server_response(response)
E                   199 json_string = response.json().get("json_string")
E               
E               File ~/edsl/edsl/coop/coop.py:84, in Coop._send_server_request(self, uri, method, payload, params, timeout)
E                    82         raise Exception(f"Invalid {method=}.")
E                    83 except requests.ConnectionError:
E               ---> 84     raise requests.ConnectionError("Could not connect to the server.")
E                    86 return response
E               
E               ConnectionError: Could not connect to the server.

integration/active/test_notebooks.py:29: AssertionError
------------------------------------------------------ Captured stdout call -------------------------------------------------------
Executing docs/notebooks/filestore_example.ipynb...
===================================================== short test summary info =====================================================
FAILED integration/active/test_notebooks.py::test_notebook_execution[docs/notebooks/filestore_example.ipynb] - AssertionError: Execution error in docs/notebooks/filestore_example.ipynb: An error occurred while executing the following cell:
================================================ 1 failed, 120 deselected in 2.27s ================================================
make: *** [test-notebooks] Error 1

[johnjosephhorton]

This is my current list out of 50 that are failing - ignore the technical analysis one:

integration/active/test_notebooks.py::test_notebook_execution[docs/notebooks/ces_agents.ipynb] FAILED                                                                          [ 12%]
integration/active/test_notebooks.py::test_notebook_execution[docs/notebooks/cheatsheet_scenarios.ipynb] FAILED                                                                [ 16%]
integration/active/test_notebooks.py::test_notebook_execution[docs/notebooks/edsl_with_cloud_providers.ipynb] FAILED                                                           [ 42%]
integration/active/test_notebooks.py::test_notebook_execution[docs/notebooks/edsl_with_offline_inference_services.ipynb] FAILED                                                [ 44%]
integration/active/test_notebooks.py::test_notebook_execution[docs/notebooks/google_form_to_edsl.ipynb] FAILED                                                                 [ 58%]
integration/active/test_notebooks.py::test_notebook_execution[docs/notebooks/model_walkoff.ipynb] FAILED                                                                       [ 66%]
integration/active/test_notebooks.py::test_notebook_execution[docs/notebooks/nps_survey.ipynb] FAILED                                                                          [ 68%]
integration/active/test_notebooks.py::test_notebook_execution[docs/notebooks/question_loop_scenarios.ipynb] FAILED                                                             [ 74%]
integration/active/test_notebooks.py::test_notebook_execution[docs/notebooks/technical_analysis.ipynb] FAILED                                                                  [ 94%]