search

dgunning / edgartools

Navigate SEC Edgar data in Python
MIT License
516 stars 101 forks source link

How to use local storage storage #117

Closed rpayanm closed 1 month ago

rpayanm commented 1 month ago

Hi there,

Thank you for this amazing lib.

I was trying to use local storage, I did this:

from edgar import *
set_identity("Name user@example.com")
download_edgar_data()

It downloaded the files.

then I disabled my wifi and ran this:

from edgar import *

use_local_storage()

# Tell the SEC who you are
# set_identity("Name user@example.com")

# download_edgar_data()

filings = Company("NFLX").get_filings(form="10-Q").latest(1)
filings.view()

I received this error:

Traceback (most recent call last):
  File "/home/user/PycharmProjects/fin-data/app.py", line 9, in <module>
    filings = Company("NFLX").get_filings(form="10-Q").latest(1)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/edgar/entities.py", line 774, in get_entity
    return EntityData.for_ticker(entity_identifier, include_old_filings=include_old_filings)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/edgar/entities.py", line 447, in for_ticker
    cik = find_cik(ticker)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/edgar/reference/tickers.py", line 131, in find_cik
    cik = find_company_cik(ticker)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/edgar/reference/tickers.py", line 118, in find_company_cik
    lookup = get_company_cik_lookup()
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/edgar/reference/tickers.py", line 51, in get_company_cik_lookup
    df = get_cik_tickers()
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/edgar/reference/tickers.py", line 37, in get_cik_tickers
    source = StringIO(download_file("https://www.sec.gov/include/ticker.txt", as_text=True))
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/edgar/httprequests.py", line 395, in download_file
    response = get_with_retry(url=url)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/stamina/_core.py", line 726, in sync_inner
    for attempt in retry_ctx.with_name(  # noqa: RET503
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/stamina/_core.py", line 508, in __iter__
    for r in _t.Retrying(
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 435, in __iter__
    do = self.iter(retry_state=retry_state)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 368, in iter
    result = action(retry_state)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 390, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/stamina/_core.py", line 730, in sync_inner
    return wrapped(*args, **kw)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/edgar/httprequests.py", line 159, in wrapper
    raise IdentityNotSetException("User-Agent identity is not set")
edgar.httprequests.IdentityNotSetException: User-Agent identity is not set
  1. Why do I need to set my User-Agent identity if I am using the local storage?

Next, I set my User-Agent identity:

from edgar import *

use_local_storage()

# Tell the SEC who you are
set_identity("Name user@example.com")

# download_edgar_data()

filings = Company("NFLX").get_filings(form="10-Q").latest(1)
filings.view()

But, I received this error:

[21:20:54] INFO     Identity of the Edgar REST client set to [Name user@example.com]                                                                                                                                 core.py:158
           WARNING  stamina.retry_scheduled                                                                                                                                                                           _logging.py:23
           WARNING  stamina.retry_scheduled                                                                                                                                                                           _logging.py:23
[21:20:55] WARNING  stamina.retry_scheduled                                                                                                                                                                           _logging.py:23
[21:20:56] WARNING  stamina.retry_scheduled                                                                                                                                                                           _logging.py:23
[21:20:57] WARNING  stamina.retry_scheduled                                                                                                                                                                           _logging.py:23
Traceback (most recent call last):
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 72, in map_httpcore_exceptions
    yield
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 236, in handle_request
    resp = self._pool.handle_request(req)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
    raise exc from None
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
    raise exc
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
    stream = self._connect(request)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 122, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
    with map_exceptions(exc_map):
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: [Errno -2] Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/PycharmProjects/fin-data/app.py", line 9, in <module>
    filings = Company("NFLX").get_filings(form="10-Q").latest(1)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/edgar/entities.py", line 774, in get_entity
    return EntityData.for_ticker(entity_identifier, include_old_filings=include_old_filings)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/edgar/entities.py", line 447, in for_ticker
    cik = find_cik(ticker)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/edgar/reference/tickers.py", line 131, in find_cik
    cik = find_company_cik(ticker)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/edgar/reference/tickers.py", line 118, in find_company_cik
    lookup = get_company_cik_lookup()
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/edgar/reference/tickers.py", line 51, in get_company_cik_lookup
    df = get_cik_tickers()
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/edgar/reference/tickers.py", line 37, in get_cik_tickers
    source = StringIO(download_file("https://www.sec.gov/include/ticker.txt", as_text=True))
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/edgar/httprequests.py", line 395, in download_file
    response = get_with_retry(url=url)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/stamina/_core.py", line 726, in sync_inner
    for attempt in retry_ctx.with_name(  # noqa: RET503
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/stamina/_core.py", line 508, in __iter__
    for r in _t.Retrying(
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 435, in __iter__
    do = self.iter(retry_state=retry_state)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 368, in iter
    result = action(retry_state)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 410, in exc_check
    raise retry_exc.reraise()
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 183, in reraise
    raise self.last_attempt.result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/stamina/_core.py", line 730, in sync_inner
    return wrapped(*args, **kw)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/edgar/httprequests.py", line 165, in wrapper
    return func(url, identity=identity, identity_callable=identity_callable, *args, **kwargs)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/edgar/httprequests.py", line 132, in wrapper
    result = func(*args, **kwargs)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/edgar/httprequests.py", line 190, in get_with_retry
    response = client.get(url, **kwargs)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1066, in get
    return self.request(
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/httpx/_client.py", line 837, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/httpx/_client.py", line 926, in send
    response = self._send_handling_auth(
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/httpx/_client.py", line 954, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/httpx/_client.py", line 991, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1027, in _send_single_request
    response = transport.handle_request(request)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 235, in handle_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/user/PycharmProjects/fin-data/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 89, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: [Errno -2] Name or service not known
  1. What am I doing wrong?

Thanks in advance.

dgunning commented 1 month ago

Hey, you are also missing the use_local_storage() call

use_local_storage()
download_edgar_data()

Currently local storage relates to company facts and company filings. Calling Company("NFLX") makes a call to the SEC's Get Company Tickers endpoint. The plan is to expand what's included in local storage in the next few weeks.