Make cronjob for Forms 3, 4 and 5 tables

[bdcallen]

@iangow In addition to the main edgar cronjob, we should make a cronjob for the tables we made for Forms 3, 4, and 5. I think I already have a basic bash script ready to go, probably just have to add some of the shebang lines to programs to make them executable by cron.

[bdcallen]

@iangow I'm running the program to update these tables through my RStudio at the moment. Will set up the cronjob afterwards

[bdcallen]

@iangow I have now run the update, and now set a job on my crontab to be done for this at midnight daily

26 17 * * * $CODE_DIR/./asx_prev_day_cronjob.sh
00 21 * * * $EDGAR_CODE_DIR/./update_edgar.sh
00 0 * * * $EDGAR_CODE_DIR/./update_forms_345_tables.sh # forms3435 cronjob
00 6 * * 5 $ABN_LOOKUP_DIR/./abn_lookup_cronjob.sh
00 3 * * 4 $ASIC_DIR/./asic_bulk_extract_cronjob.sh

I will keep an eye on the dead.letter file for a few days to see if the cronjob is working

[bdcallen]

@iangow

Running get_filings.R ...
Updating data for 2019Q4...
Running get_accession_nos.R ...
Running get_filer_ciks.R ...
Running get_item_nos.R ...
Processing batch 1 of 3 ... 52.778 seconds
Processing batch 2 of 3 ... 46.165 seconds
Processing batch 3 of 3 ... 108.113 seconds
Running get_item_no_desc.R ...
Running scrape_filing_docs.R ...
Processing batch 1 
Writing data ...
5065.518 seconds
Processing batch 2 
Writing data ...
77.9691 seconds
Processing batch 3 
Writing data ...
83.76994 seconds
Processing batch 4 
Writing data ...
504.3693 seconds

Loading required package: DBI
Loading required package: bitops

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

Attaching package: ‘lubridate’

The following object is masked from ‘package:base’:

    date

Loading required package: xml2

Attaching package: ‘rvest’

The following object is masked from ‘package:XML’:

    xml

[1] "Total time taken: \n"
   user  system elapsed 
536.642 101.986 108.302 
[1] "Number of full successes: \n"
[1] 2163
[1] "Number of filings processed: \n"
[1] 2163
[1] TRUE

Attaching package: ‘lubridate’

The following object is masked from ‘package:base’:

    date

Attaching package: ‘dplyr’

The following objects are masked from ‘package:lubridate’:

    intersect, setdiff, union

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

[1] TRUE

After making the bash script executable and doing chmod u+x, it seems the forms 3, 4, 5 cron is working, going by the latest output written to dead.letter. So I am going to close this.

[iangow]

How long did it take to run?

Sent from my iPhone

On 19 Dec 2019, at 15:09, Benjamin Callen notifications@github.com wrote:

 Closed #67.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.