search

toddwschneider / sec-13f-filings

A nicer way to view SEC 13F filings data
https://13f.info
MIT License
248 stars 52 forks source link

No holdings imported #6

Closed priyansh-anand closed 1 year ago

priyansh-anand commented 1 year ago

I am following the readme to get the minimal holdings in the database. Here are the commands executed: image

After this is done executing, the holdings & aggregated holdings table is still empty but I can see some errors in the delayed jobs table: image

One of the value from the last_error column:

undefined method `text' for nil:NilClass
/app/lib/sec_client.rb:111:in `parse_primary_doc_xml'
/app/models/thirteen_f.rb:337:in `parsed_primary_doc'
/app/models/thirteen_f.rb:171:in `cache_attributes_from_primary_doc'
/app/models/thirteen_f.rb:140:in `process!'
/usr/local/bundle/gems/delayed_job-4.1.9/lib/delayed/performable_method.rb:26:in `perform'
/usr/local/bundle/gems/delayed_job-4.1.9/lib/delayed/backend/base.rb:81:in `block in invoke_job'
/usr/local/bundle/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:61:in `block in initialize'
/usr/local/bundle/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:66:in `execute'
/usr/local/bundle/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/usr/local/bundle/gems/delayed_job-4.1.9/lib/delayed/backend/base.rb:78:in `invoke_job'
/usr/local/bundle/gems/delayed_job-4.1.9/lib/delayed/worker.rb:230:in `block (2 levels) in run'
/usr/local/lib/ruby/3.0.0/timeout.rb:97:in `block in timeout'
/usr/local/lib/ruby/3.0.0/timeout.rb:107:in `timeout'
/usr/local/bundle/gems/delayed_job-4.1.9/lib/delayed/worker.rb:230:in `block in run'
/usr/local/lib/ruby/3.0.0/benchmark.rb:308:in `realtime'
/usr/local/bundle/gems/delayed_job-4.1.9/lib/delayed/worker.rb:229:in `run'
/usr/local/bundle/gems/delayed_job-4.1.9/lib/delayed/worker.rb:312:in `block in reserve_and_run_one_job'
/usr/local/bundle/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:61:in `block in initialize'
/usr/local/bundle/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:66:in `execute'
/usr/local/bundle/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/usr/local/bundle/gems/delayed_job-4.1.9/lib/delayed/worker.rb:312:in `reserve_and_run_one_job'
/usr/local/bundle/gems/delayed_job-4.1.9/lib/delayed/worker.rb:213:in `block in work_off'
/usr/local/bundle/gems/delayed_job-4.1.9/lib/delayed/worker.rb:212:in `times'
/usr/local/bundle/gems/delayed_job-4.1.9/lib/delayed/worker.rb:212:in `work_off'
/app/lib/minimal_db_seeder.rb:36:in `seed_minimal_db!'
/lib/tasks/import_filings.rake:4:in `block (2 levels) in <main>'
/usr/local/bundle/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/usr/local/bundle/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/usr/local/bundle/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/usr/local/bundle/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/usr/local/bundle/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/usr/local/bundle/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/usr/local/bundle/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/usr/local/bundle/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/usr/local/bundle/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/usr/local/bundle/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/usr/local/bundle/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/usr/local/bundle/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/usr/local/bundle/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/usr/local/bundle/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/usr/local/bundle/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/usr/local/bundle/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/usr/local/bundle/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/usr/local/bundle/bin/rake:23:in `load'
/usr/local/bundle/bin/rake:23:in `<top (required)>'
/usr/local/bundle/gems/bundler-2.2.25/lib/bundler/cli/exec.rb:63:in `load'
/usr/local/bundle/gems/bundler-2.2.25/lib/bundler/cli/exec.rb:63:in `kernel_load'
/usr/local/bundle/gems/bundler-2.2.25/lib/bundler/cli/exec.rb:28:in `run'
/usr/local/bundle/gems/bundler-2.2.25/lib/bundler/cli.rb:481:in `exec'
/usr/local/bundle/gems/bundler-2.2.25/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/bundle/gems/bundler-2.2.25/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/bundle/gems/bundler-2.2.25/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/usr/local/bundle/gems/bundler-2.2.25/lib/bundler/cli.rb:31:in `dispatch'
/usr/local/bundle/gems/bundler-2.2.25/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/usr/local/bundle/gems/bundler-2.2.25/lib/bundler/cli.rb:25:in `start'
/usr/local/bundle/gems/bundler-2.2.25/exe/bundle:49:in `block in <top (required)>'
/usr/local/bundle/gems/bundler-2.2.25/lib/bundler/friendly_errors.rb:128:in `with_friendly_errors'
/usr/local/bundle/gems/bundler-2.2.25/exe/bundle:37:in `<top (required)>'
/usr/local/bundle/bin/bundle:23:in `load'
/usr/local/bundle/bin/bundle:23:in `<main>'`

Also, these are the 13Fs that are left in the thirteen_fs table: image

toddwschneider commented 1 year ago

Hi @priyansh-anand thanks for reporting. I think it's probably an issue with the SEC website blocking due to missing user agent: https://www.sec.gov/os/webmaster-faq#code-support

I will take a look and see if I can fix

toddwschneider commented 1 year ago

@priyansh-anand I believe this should be fixed by this pull request: https://github.com/toddwschneider/sec-13f-filings/pull/7

Note that you'll have to create a .env file locally and set your own environment variable. Give it a shot and let me know if any issues. I'm going to close this issue, but feel free to reopen if you still have the same problem. Thanks