search

iangow / se_core

Core code for StreetEvents data
7 stars 5 forks source link

Create CRON job for se_core #7

Closed iangow closed 3 years ago

iangow commented 5 years ago 
#!/usr/bin/env bash
export PGHOST=localhost
export PGDATABASE=crsp
export PGPASSWORD=xxxx
export PGUSER=igow
export CODE_DIR=/home/igow/git/se_core
cd $CODE_DIR
$CODE_DIR/update_se.sh
~
iangow commented 5 years ago

Added above to se_core_update and made crontab -e entry. Need to check that this is working. If it is, should see more rows tomorrow but no update after running this code:

> system("./update_se.sh")
Importing call_files
localhost
Updating data on localhost 
Importing calls
Loading required package: DBI
Updating data on localhost 
[1] 824439
Importing speaker_data
Importing speaker data.
   user  system elapsed 
  0.195   0.012  64.212
iangow commented 5 years ago

Actually needs to get XML files too.

iangow commented 5 years ago

I set up ssmtp to get error messages sent to my email address:

 sudo apt install ssmtp

I then followed instructions found here.

iangow commented 4 years ago

@bdcallen I think we can close this with #7 once that is dealt with. It's important to check that the cron job is downloading incremental files.

bdcallen commented 4 years ago

@iangow 

crsp=> select relname, description from pg_description
join pg_class on pg_description.objoid = pg_class.oid
where relname IN (SELECT table_name FROM information_schema.tables 
WHERE table_schema = 'streetevents');
        relname        |                                        description                                         
-----------------------+--------------------------------------------------------------------------------------------
 company_ids           | CREATED USING create_company_ids.R from GitHub iangow/se_core ON 2020-01-08 03:19:50
 calls_raw             | UPDATED USING import_calls_raw.R from GitHub iangow/se_core ON 2020-01-07 16:19:31
 calls_test            | CREATED USING create_calls_test.R from GitHub iangow/se_core ON 2019-12-06 17:12:29
 crsp_link             | CREATED USING create_crsp_link.R ON 2020-01-08 10:35:32
 manual_permno_matches | CREATED USING import_manual_permno_matches.R ON 2020-01-08 10:25:16 ON 2020-01-08 10:25:16
 selected_calls        | CREATED USING iangow/se_core/create_selected_calls.R ON 2020-01-08 03:20:05
 calls                 | CREATED USING create_calls.R from GitHub iangow/se_core ON 2020-01-08 03:19:46
(7 rows)
bdcallen commented 4 years ago

@iangow Tried running update_se.sh just now and got

bdcallen@igow-z640:~/se_core$ ./update_se.sh
Importing call_files
10.101.13.99
Updating data on 10.101.13.99 
Importing calls
Loading required package: DBI
Updating data on 10.101.13.99 
Warning messages:
1: `lang_name()` is deprecated as of rlang 0.2.0.
Please use `call_name()` instead.
This warning is displayed once per session. 
2: `lang()` is deprecated as of rlang 0.2.0.
Please use `call2()` instead.
This warning is displayed once per session. 
[1] 1318085
Error in (function (classes, fdef, mtable)  : 
  unable to find an inherited method for function ‘dbExecute’ for signature ‘"character", "missing"’
Calls: dbExecute -> <Anonymous>
Execution halted
Warning messages:
1: `lang_name()` is deprecated as of rlang 0.2.0.
Please use `call_name()` instead.
This warning is displayed once per session. 
2: `lang()` is deprecated as of rlang 0.2.0.
Please use `call2()` instead.
This warning is displayed once per session. 
Importing speaker_data
Importing speaker data.
   user  system elapsed 
  0.215   0.008  59.453
bdcallen commented 4 years ago 
bdcallen@igow-z640:/home/shared/uploads$ ls -hlt
total 488M
-rw-r--r-- 1 igow mccgr  44K Jan  8 02:50 12976736_T.xml
-rw-r--r-- 1 igow mccgr  60K Jan  8 02:47 12957278_T.xml
-rw-r--r-- 1 igow mccgr  31K Jan  8 02:40 12936603_T.xml
-rw-r--r-- 1 igow mccgr  71K Jan  8 02:40 12769260_T.xml
-rw-r--r-- 1 igow mccgr  42K Jan  8 02:39 12905763_T.xml
-rw-r--r-- 1 igow mccgr 129K Jan  8 02:30 12872322_T.xml
-rw-r--r-- 1 igow mccgr  27K Jan  8 02:21 12412761_T.xml
-rw-r--r-- 1 igow mccgr  42K Jan  8 02:18 12954574_T.xml
-rw-r--r-- 1 igow mccgr  34K Jan  8 02:16 12963528_T.xml
-rw-r--r-- 1 igow mccgr  63K Jan  8 02:14 12984173_T.xml
-rw-r--r-- 1 igow mccgr  27K Jan  8 02:12 12968998_T.xml
-rw-r--r-- 1 igow mccgr  48K Jan  8 02:04 12968997_T.xml
-rw-r--r-- 1 igow mccgr  23K Jan  8 02:04 12954773_T.xml
-rw-r--r-- 1 igow mccgr  68K Jan  8 02:03 12983188_T.xml
-rw-r--r-- 1 igow mccgr  45K Jan  8 01:58 12958607_T.xml
-rw-r--r-- 1 igow mccgr  43K Jan  8 01:57 12815025_T.xml
-rw-r--r-- 1 igow mccgr  53K Jan  8 01:55 12952468_T.xml
-rw-r--r-- 1 igow mccgr  35K Jan  8 01:52 12983349_T.xml
-rw-r--r-- 1 igow mccgr  52K Jan  8 01:47 12172359_T.xml
-rw-r--r-- 1 igow mccgr  64K Jan  8 01:41 12954109_T.xml
-rw-r--r-- 1 igow mccgr  48K Jan  8 01:41 12964938_T.xml
-rw-r--r-- 1 igow mccgr  28K Jan  8 01:38 12804367_T.xml
-rw-r--r-- 1 igow mccgr  33K Jan  8 01:36 12521372_T.xml
-rw-r--r-- 1 igow mccgr  25K Jan  8 01:35 12960362_T.xml
-rw-r--r-- 1 igow mccgr  61K Jan  8 01:33 12984141_T.xml
-rw-r--r-- 1 igow mccgr  43K Jan  8 01:22 12746668_T.xml
-rw-r--r-- 1 igow mccgr  50K Jan  8 01:14 12982347_T.xml
-rw-r--r-- 1 igow mccgr  33K Jan  8 01:13 12884684_T.xml
-rw-r--r-- 1 igow mccgr  48K Jan  8 01:12 12675193_T.xml
-rw-r--r-- 1 igow mccgr  58K Jan  8 01:09 12536189_T.xml
-rw-r--r-- 1 igow mccgr 120K Jan  8 01:09 12909281_T.xml
-rw-r--r-- 1 igow mccgr  49K Jan  8 01:09 12297383_T.xml
-rw-r--r-- 1 igow mccgr  46K Jan  8 01:09 12806977_T.xml
-rw-r--r-- 1 igow mccgr 119K Jan  8 01:09 12825808_T.xml
-rw-r--r-- 1 igow mccgr  47K Jan  8 01:09 12944853_T.xml
-rw-r--r-- 1 igow mccgr  34K Jan  8 01:09 12954369_T.xml
-rw-r--r-- 1 igow mccgr  44K Jan  8 01:09 12955066_T.xml
-rw-r--r-- 1 igow mccgr  34K Jan  8 01:08 12764234_T.xml
-rw-r--r-- 1 igow mccgr  24K Jan  8 00:53 12789550_T.xml
-rw-r--r-- 1 igow mccgr  40K Jan  8 00:49 12964130_T.xml
-rw-r--r-- 1 igow mccgr  38K Jan  8 00:32 12962019_T.xml

...

drwxr-xr-x 2 igow mccgr 4.0K Oct  9 03:16 backfill
-rw-r--r-- 1 igow mccgr  40K Oct  9 02:55 12634716_T.xml
-rw-r--r-- 1 igow mccgr  48K Oct  9 02:55 12632991_T.xml
-rw-r--r-- 1 igow mccgr  68K Oct  9 02:55 12609161_T.xml
-rw-r--r-- 1 igow mccgr  45K Oct  9 02:55 12627919_T.xml
-rw-r--r-- 1 igow mccgr  44K Oct  9 02:55 12704352_T.xml
-rw-r--r-- 1 igow mccgr  31K Oct  9 02:55 12748053_T.xml
-rw-r--r-- 1 igow mccgr 158K Oct  9 01:11 12734118_T.xml
-rw-r--r-- 1 igow mccgr  64K Oct  9 01:10 12620175_T.xml
-rw-r--r-- 1 igow mccgr  19K Oct  9 01:10 12605710_T.xml
-rw-r--r-- 1 igow mccgr  46K Oct  9 00:48 12778281_T.xml
-rw-r--r-- 1 igow mccgr 167K Oct  8 23:20 12718915_T.xml
-rw-r--r-- 1 igow mccgr  20K Oct  8 23:20 12616234_T.xml
-rw-r--r-- 1 igow mccgr  57K Oct  8 21:33 12607948_T.xml
-rw-r--r-- 1 igow mccgr  44K Oct  8 21:33 12764189_T.xml
-rw-r--r-- 1 igow mccgr  34K Oct  8 19:16 12729137_T.xml
-rw-r--r-- 1 igow mccgr  55K Oct  8 19:16 12612002_T.xml
-rw-r--r-- 1 igow mccgr  91K Oct  8 19:16 12750785_T.xml
-rw-r--r-- 1 igow mccgr  44K Oct  8 19:16 12764484_T.xml
-rw-r--r-- 1 igow mccgr  37K Oct  8 17:15 12558373_T.xml
-rw-r--r-- 1 igow mccgr  75K Oct  8 17:15 12629878_T.xml
-rw-r--r-- 1 igow mccgr  74K Oct  8 17:15 12631075_T.xml
-rw-r--r-- 1 igow mccgr  36K Oct  8 17:14 12645492_T.xml
-rw-r--r-- 1 igow mccgr  80K Oct  8 15:35 12624681_T.xml
-rw-r--r-- 1 igow mccgr  42K Oct  8 15:35 12493615_T.xml
-rw-r--r-- 1 igow mccgr  46K Oct  8 15:35 12220475_T.xml
-rw-r--r-- 1 igow mccgr  66K Oct  8 15:35 12625729_T.xml
-rw-r--r-- 1 igow mccgr 6.7K Oct  8 15:35 12676110_T.xml
-rw-r--r-- 1 igow mccgr 104K Oct  8 15:35 12712480_T.xml
-rw-r--r-- 1 igow mccgr  58K Oct  8 15:35 12627030_T.xml
-rw-r--r-- 1 igow mccgr  71K Oct  8 13:54 12220936_T.xml
-rw-r--r-- 1 igow mccgr  29K Oct  8 13:54 12220774_T.xml
-rw-r--r-- 1 igow mccgr  74K Oct  8 13:54 12604768_T.xml
-rw-r--r-- 1 igow mccgr  39K Oct  8 13:54 12719085_T.xml
-rw-r--r-- 1 igow mccgr  36K Oct  8 12:05 12608430_T.xml
-rw-r--r-- 1 igow mccgr  96K Oct  8 12:05 12669954_T.xml
-rw-r--r-- 1 igow mccgr  59K Oct  8 12:05 12698673_T.xml
-rw-r--r-- 1 igow mccgr  38K Oct  8 12:05 12718938_T.xml
-rw-r--r-- 1 igow mccgr  32K Oct  8 12:05 12765831_T.xml
-rw-r--r-- 1 igow mccgr  47K Oct  8 12:05 12617047_T.xml
-rw-r--r-- 1 igow mccgr  42K Oct  8 12:05 12703380_T.xml
-rw-r--r-- 1 igow mccgr  60K Oct  8 12:05 12719106_T.xml
-rw-r--r-- 1 igow mccgr  28K Oct  8 12:05 12765357_T.xml
-rw-r--r-- 1 igow mccgr  59K Oct  8 10:29 12771745_T.xml
-rw-r--r-- 1 igow mccgr  64K Oct  8 10:06 11771508_T.xml
-rw-r--r-- 1 igow mccgr  21K Oct  8 10:06 12183861_T.xml
-rw-r--r-- 1 igow mccgr  37K Oct  8 10:06 12604428_T.xml
-rw-r--r-- 1 igow mccgr  34K Oct  8 10:06 12703384_T.xml
-rw-r--r-- 1 igow mccgr  68K Oct  8 10:06 12715153_T.xml
-rw-r--r-- 1 igow mccgr 128K Oct  8 08:15 12597120_T.xml
-rw-r--r-- 1 igow mccgr  48K Oct  8 08:15 12614831_T.xml
-rw-r--r-- 1 igow mccgr  44K Oct  8 08:15 12651230_T.xml
-rw-r--r-- 1 igow mccgr  50K Oct  8 08:15 12702491_T.xml
-rw-r--r-- 1 igow mccgr  38K Oct  8 08:15 12717071_T.xml
-rw-r--r-- 1 igow mccgr  93K Oct  8 08:15 12719105_T.xml
-rw-r--r-- 1 igow mccgr  33K Oct  8 08:15 12748739_T.xml
-rw-r--r-- 1 igow mccgr  75K Oct  8 08:15 12613678_T.xml
-rw-r--r-- 1 igow mccgr  52K Oct  8 08:15 12706979_T.xml
-rw-r--r-- 1 igow mccgr  31K Oct  8 08:15 12711173_T.xml
-rw-r--r-- 1 igow mccgr  55K Oct  8 08:15 12765380_T.xml
-rw-r--r-- 1 igow mccgr  24K Oct  8 08:15 12571289_T.xml
-rw-r--r-- 1 igow mccgr  35K Oct  8 08:15 12698572_T.xml
-rw-r--r-- 1 igow mccgr 126K Oct  8 06:16 12651880_T.xml
-rw-r--r-- 1 igow mccgr  55K Oct  8 06:16 12719974_T.xml
-rw-r--r-- 1 igow mccgr  80K Oct  8 04:43 12587056_T.xml
-rw-r--r-- 1 igow mccgr 132K Oct  8 04:16 12583484_T.xml
-rw-r--r-- 1 igow mccgr 260K Oct  8 04:16 12706996_T.xml
-rw-r--r-- 1 igow mccgr 298K Oct  8 04:16 12736778_T.xml
-rw-r--r-- 1 igow mccgr  74K Oct  8 04:15 12682675_T.xml
-rw-r--r-- 1 igow mccgr  37K Oct  8 04:15 12702493_T.xml
-rw-r--r-- 1 igow mccgr  31K Oct  8 04:15 12711897_T.xml
-rw-r--r-- 1 igow mccgr 254K Oct  8 02:27 12683059_T.xml
-rw-r--r-- 1 igow mccgr  32K Oct  8 02:27 12725425_T.xml
-rw-r--r-- 1 igow mccgr  77K Oct  8 02:27 12678060_T.xml
-rw-r--r-- 1 igow mccgr  53K Oct  8 02:27 12711339_T.xml
-rw-r--r-- 1 igow mccgr  41K Oct  8 02:27 12719988_T.xml
-rw-r--r-- 1 igow mccgr  56K Oct  8 02:27 12736252_T.xml
-rw-r--r-- 1 igow mccgr 178K Oct  8 02:27 12765755_T.xml
-rw-r--r-- 1 igow mccgr  64K Oct  8 00:35 12616267_T.xml
-rw-r--r-- 1 igow mccgr  37K Oct  8 00:35 12627622_T.xml
-rw-r--r-- 1 igow mccgr  40K Oct  8 00:35 12634856_T.xml
-rw-r--r-- 1 igow mccgr  62K Oct  7 22:38 12747047_T.xml
-rw-r--r-- 1 igow mccgr  20K Oct  7 22:38 12760304_T.xml
-rw-r--r-- 1 igow mccgr  44K Oct  7 20:50 12740575_T.xml
-rw-r--r-- 1 igow mccgr  13K Oct  7 19:07 12765400_T.xml
-rw-r--r-- 1 igow mccgr 184K Oct  7 18:58 12750962_T.xml
-rw-r--r-- 1 igow mccgr  35K Oct  7 17:09 12620597_T.xml
-rw-r--r-- 1 igow mccgr  58K Oct  7 17:09 12632235_T.xml
-rw-r--r-- 1 igow mccgr  26K Oct  7 15:18 12733964_T.xml
-rw-r--r-- 1 igow mccgr  90K Oct  6 13:21 12457695_T.xml
-rw-r--r-- 1 igow mccgr  61K Oct  6 13:21 12549313_T.xml
-rw-r--r-- 1 igow mccgr  40K Oct  6 13:20 12532986_T.xml
-rw-r--r-- 1 igow mccgr 226K Oct  6 11:25 12347044_T.xml
-rw-r--r-- 1 igow mccgr  58K Oct  6 11:25 12610174_T.xml
-rw-r--r-- 1 igow mccgr  34K Oct  6 11:25 12631226_T.xml
-rw-r--r-- 1 igow mccgr  38K Oct  6 11:25 12709496_T.xml
-rw-r--r-- 1 igow mccgr 135K Oct  6 11:25 12712482_T.xml
-rw-r--r-- 1 igow mccgr  36K Oct  6 09:27 12674288_T.xml
-rw-r--r-- 1 igow mccgr  33K Oct  6 09:27 12677092_T.xml
-rw-r--r-- 1 igow mccgr  62K Oct  6 07:32 12697269_T.xml
-rw-r--r-- 1 igow mccgr  45K Oct  6 07:32 12697782_T.xml
-rw-r--r-- 1 igow mccgr 103K Oct  6 07:32 12709692_T.xml
-rw-r--r-- 1 igow mccgr  17K Oct  6 05:39 12222821_T.xml
-rw-r--r-- 1 igow mccgr 113K Oct  6 05:39 12674190_T.xml
-rw-r--r-- 1 igow mccgr  45K Oct  5 18:51 12281605_T.xml
-rw-r--r-- 1 igow mccgr  57K Oct  5 18:51 12719285_T.xml
-rw-r--r-- 1 igow mccgr  42K Oct  5 18:51 12734150_T.xml
-rw-r--r-- 1 igow mccgr 158K Oct  5 16:55 12553796_T.xml
-rw-r--r-- 1 igow mccgr  44K Oct  5 16:55 12720607_T.xml
-rw-r--r-- 1 igow mccgr 141K Oct  5 16:55 12725416_T.xml
-rw-r--r-- 1 igow mccgr  41K Oct  5 16:55 12726179_T.xml
-rw-r--r-- 1 igow mccgr  48K Oct  5 14:56 12726477_T.xml
-rw-r--r-- 1 igow mccgr  69K Oct  5 14:56 12728273_T.xml
-rw-r--r-- 1 igow mccgr  91K Oct  5 13:00 12621163_T.xml
-rw-r--r-- 1 igow mccgr  56K Oct  5 13:00 12632344_T.xml
-rw-r--r-- 1 igow mccgr  64K Oct  5 13:00 12633163_T.xml
-rw-r--r-- 1 igow mccgr  54K Oct  5 13:00 12642313_T.xml
-rw-r--r-- 1 igow mccgr  42K Oct  5 13:00 12669198_T.xml
-rw-r--r-- 1 igow mccgr  80K Oct  5 11:03 12647622_T.xml
-rw-r--r-- 1 igow mccgr  62K Oct  5 11:03 12620239_T.xml
-rw-r--r-- 1 igow mccgr  70K Oct  5 11:03 12673840_T.xml
-rw-r--r-- 1 igow mccgr  71K Oct  5 11:03 12707004_T.xml
-rw-r--r-- 1 igow mccgr  42K Oct  5 11:03 12708179_T.xml
-rw-r--r-- 1 igow mccgr  74K Oct  5 11:03 12719326_T.xml
-rw-r--r-- 1 igow mccgr  57K Oct  5 09:05 12142495_T.xml
-rw-r--r-- 1 igow mccgr  67K Oct  5 09:05 12592486_T.xml
-rw-r--r-- 1 igow mccgr  29K Oct  5 09:05 12713957_T.xml
-rw-r--r-- 1 igow mccgr  38K Oct  5 09:05 5603205_T.xml
-rw-r--r-- 1 igow mccgr  87K Oct  5 09:05 12603877_T.xml
-rw-r--r-- 1 igow mccgr 109K Oct  5 09:05 12627401_T.xml
-rw-r--r-- 1 igow mccgr  75K Oct  5 09:05 12642068_T.xml
-rw-r--r-- 1 igow mccgr  45K Oct  5 09:05 12718063_T.xml
-rw-r--r-- 1 igow mccgr  87K Oct  5 07:05 12215571_T.xml
-rw-r--r-- 1 igow mccgr  35K Oct  5 07:05 12613946_T.xml
-rw-r--r-- 1 igow mccgr  45K Oct  5 07:05 12763826_T.xml
-rw-r--r-- 1 igow mccgr  54K Oct  5 07:05 12638757_T.xml
-rw-r--r-- 1 igow mccgr  46K Oct  5 07:05 5314330_T.xml
-rw-r--r-- 1 igow mccgr  76K Oct  5 07:05 12634818_T.xml
-rw-r--r-- 1 igow mccgr  37K Oct  5 07:05 12702468_T.xml
-rw-r--r-- 1 igow mccgr  41K Oct  5 07:05 12710000_T.xml
-rw-r--r-- 1 igow mccgr  66K Oct  5 05:05 12633017_T.xml
-rw-r--r-- 1 igow mccgr  55K Oct  5 05:05 12638346_T.xml
-rw-r--r-- 1 igow mccgr  58K Oct  5 05:05 12677437_T.xml
-rw-r--r-- 1 igow mccgr  39K Oct  5 05:05 12734562_T.xml
-rw-r--r-- 1 igow mccgr  49K Oct  5 05:05 12702473_T.xml
-rw-r--r-- 1 igow mccgr  65K Oct  5 05:05 12703036_T.xml
-rw-r--r-- 1 igow mccgr  41K Oct  5 05:05 12708403_T.xml
-rw-r--r-- 1 igow mccgr  34K Oct  5 05:05 12765365_T.xml
-rw-r--r-- 1 igow mccgr  26K Oct  5 05:05 5030775_T.xml
-rw-r--r-- 1 igow mccgr  51K Oct  5 05:05 12683953_T.xml
-rw-r--r-- 1 igow mccgr  34K Oct  5 05:05 12691211_T.xml
-rw-r--r-- 1 igow mccgr  60K Oct  5 03:06 12764028_T.xml
-rw-r--r-- 1 igow mccgr  40K Oct  5 03:05 12734519_T.xml
-rw-r--r-- 1 igow mccgr  54K Oct  5 03:05 12753252_T.xml
-rw-r--r-- 1 igow mccgr 146K Oct  5 03:05 12719810_T.xml
-rw-r--r-- 1 igow mccgr  54K Oct  5 03:05 12655692_T.xml
-rw-r--r-- 1 igow mccgr  50K Oct  5 03:05 12707471_T.xml
-rw-r--r-- 1 igow mccgr  47K Oct  5 03:05 12752645_T.xml
-rw-r--r-- 1 igow mccgr  48K Oct  5 03:05 12765626_T.xml
-rw-r--r-- 1 igow mccgr  28K Oct  5 02:26 5167812_T.xml
-rw-r--r-- 1 igow mccgr  28K Oct  5 01:05 12735449_T.xml
-rw-r--r-- 1 igow mccgr  46K Oct  5 01:05 12765688_T.xml
-rw-r--r-- 1 igow mccgr  40K Oct  5 01:04 12764152_T.xml
-rw-r--r-- 1 igow mccgr  50K Oct  4 23:11 12724597_T.xml
-rw-r--r-- 1 igow mccgr  40K Oct  4 23:11 12725800_T.xml
-rw-r--r-- 1 igow mccgr  70K Oct  4 23:11 12753710_T.xml
-rw-r--r-- 1 igow mccgr  33K Oct  4 19:56 12764638_T.xml
-rw-r--r-- 1 igow mccgr 252K Oct  4 17:21 12740317_T.xml
-rw-r--r-- 1 igow mccgr  69K Oct  4 15:30 12221059_T.xml
-rw-r--r-- 1 igow mccgr  38K Oct  4 15:30 12620607_T.xml
-rw-r--r-- 1 igow mccgr  63K Oct  4 15:30 12641056_T.xml
-rw-r--r-- 1 igow mccgr  37K Oct  4 15:30 12718897_T.xml
drwxr-xr-x 3 igow mccgr  15M Oct  4 10:12 archive_01
bdcallen@igow-z640:/home/shared/uploads$ bdcallen@igow-z640:~/se_core$ ./update_se.sh
bash: bdcallen@igow-z640:~/se_core$: No such file or directory
bdcallen@igow-z640:/home/shared/uploads$ Importing call_files
Importing: command not found
bdcallen@igow-z640:/home/shared/uploads$ 10.101.13.99
10.101.13.99: command not found
bdcallen@igow-z640:/home/shared/uploads$ Updating data on 10.101.13.99 
Updating: command not found
bdcallen@igow-z640:/home/shared/uploads$ Importing calls
Importing: command not found
bdcallen@igow-z640:/home/shared/uploads$ Loading required package: DBI
Loading: command not found
bdcallen@igow-z640:/home/shared/uploads$ Updating data on 10.101.13.99 
Updating: command not found
bdcallen@igow-z640:/home/shared/uploads$ Warning messages:
Warning: command not found
bdcallen@igow-z640:/home/shared/uploads$ 1: `lang_name()` is deprecated as of rlang 0.2.0.
bash: command substitution: line 2: syntax error: unexpected end of file
1:: command not found
bdcallen@igow-z640:/home/shared/uploads$ Please use `call_name()` instead.
bash: command substitution: line 2: syntax error: unexpected end of file
Please: command not found
bdcallen@igow-z640:/home/shared/uploads$ This warning is displayed once per session. 
This: command not found
bdcallen@igow-z640:/home/shared/uploads$ 2: `lang()` is deprecated as of rlang 0.2.0.
bash: command substitution: line 2: syntax error: unexpected end of file
2:: command not found
bdcallen@igow-z640:/home/shared/uploads$ Please use `call2()` instead.
bash: command substitution: line 2: syntax error: unexpected end of file
Please: command not found
bdcallen@igow-z640:/home/shared/uploads$ This warning is displayed once per session. 
This: command not found
bdcallen@igow-z640:/home/shared/uploads$ [1] 1318085
[1]: command not found
bdcallen@igow-z640:/home/shared/uploads$ Error in (function (classes, fdef, mtable)  : 
bash: syntax error near unexpected token `('
bdcallen@igow-z640:/home/shared/uploads$   unable to find an inherited method for function ‘dbExecute’ for signature ‘"character", "missing"’
unable: command not found
bdcallen@igow-z640:/home/shared/uploads$ Calls: dbExecute -> <Anonymous>
bash: syntax error near unexpected token `<'
bdcallen@igow-z640:/home/shared/uploads$ Execution halted
Execution: command not found
bdcallen@igow-z640:/home/shared/uploads$ Warning messages:
Warning: command not found
bdcallen@igow-z640:/home/shared/uploads$ 1: `lang_name()` is deprecated as of rlang 0.2.0.
bash: command substitution: line 2: syntax error: unexpected end of file
1:: command not found
bdcallen@igow-z640:/home/shared/uploads$ Please use `call_name()` instead.
bash: command substitution: line 2: syntax error: unexpected end of file
Please: command not found
bdcallen@igow-z640:/home/shared/uploads$ This warning is displayed once per session. 
This: command not found
bdcallen@igow-z640:/home/shared/uploads$ 2: `lang()` is deprecated as of rlang 0.2.0.
bash: command substitution: line 2: syntax error: unexpected end of file
2:: command not found
bdcallen@igow-z640:/home/shared/uploads$ Please use `call2()` instead.
bash: command substitution: line 2: syntax error: unexpected end of file
Please: command not found
bdcallen@igow-z640:/home/shared/uploads$ This warning is displayed once per session. 
This: command not found
bdcallen@igow-z640:/home/shared/uploads$ Importing speaker_data
Importing: command not found
bdcallen@igow-z640:/home/shared/uploads$ Importing speaker data.
Importing: command not found
bdcallen@igow-z640:/home/shared/uploads$    user  system elapsed 

Command 'user' not found, did you mean:

  command 'fuser' from deb psmisc
  command 'users' from deb coreutils
  command 'userv' from deb userv
  command 'iuser' from deb ipmiutil

Try: sudo apt install <deb name>

bdcallen@igow-z640:/home/shared/uploads$   0.215   0.008  59.453 
0.215: command not found
bdcallen@igow-z640:/home/shared/uploads$ bdcallen@igow-z640:~/se_core$ ./update_se.sh
bash: bdcallen@igow-z640:~/se_core$: No such file or directory
bdcallen@igow-z640:/home/shared/uploads$ Importing call_files
Importing: command not found
bdcallen@igow-z640:/home/shared/uploads$ 10.101.13.99
10.101.13.99: command not found
bdcallen@igow-z640:/home/shared/uploads$ Updating data on 10.101.13.99 
Updating: command not found
bdcallen@igow-z640:/home/shared/uploads$ Importing calls
Importing: command not found
bdcallen@igow-z640:/home/shared/uploads$ Loading required package: DBI
Loading: command not found
bdcallen@igow-z640:/home/shared/uploads$ Updating data on 10.101.13.99 
Updating: command not found
bdcallen@igow-z640:/home/shared/uploads$ Warning messages:
Warning: command not found
bdcallen@igow-z640:/home/shared/uploads$ 1: `lang_name()` is deprecated as of rlang 0.2.0.
bash: command substitution: line 2: syntax error: unexpected end of file
1:: command not found
bdcallen@igow-z640:/home/shared/uploads$ Please use `call_name()` instead.
bash: command substitution: line 2: syntax error: unexpected end of file
Please: command not found
bdcallen@igow-z640:/home/shared/uploads$ This warning is displayed once per session. 
This: command not found
bdcallen@igow-z640:/home/shared/uploads$ 2: `lang()` is deprecated as of rlang 0.2.0.
bash: command substitution: line 2: syntax error: unexpected end of file
2:: command not found
bdcallen@igow-z640:/home/shared/uploads$ Please use `call2()` instead.
bash: command substitution: line 2: syntax error: unexpected end of file
Please: command not found
bdcallen@igow-z640:/home/shared/uploads$ This warning is displayed once per session. 
This: command not found
bdcallen@igow-z640:/home/shared/uploads$ [1] 1318085
[1]: command not found
bdcallen@igow-z640:/home/shared/uploads$ Error in (function (classes, fdef, mtable)  : 
bash: syntax error near unexpected token `('
bdcallen@igow-z640:/home/shared/uploads$   unable to find an inherited method for function ‘dbExecute’ for signature ‘"character", "missing"’
unable: command not found
bdcallen@igow-z640:/home/shared/uploads$ Calls: dbExecute -> <Anonymous>
bash: syntax error near unexpected token `<'
bdcallen@igow-z640:/home/shared/uploads$ Execution halted
Execution: command not found
bdcallen@igow-z640:/home/shared/uploads$ Warning messages:
Warning: command not found
bdcallen@igow-z640:/home/shared/uploads$ 1: `lang_name()` is deprecated as of rlang 0.2.0.
bash: command substitution: line 2: syntax error: unexpected end of file
1:: command not found
bdcallen@igow-z640:/home/shared/uploads$ Please use `call_name()` instead.
bash: command substitution: line 2: syntax error: unexpected end of file
Please: command not found
bdcallen@igow-z640:/home/shared/uploads$ This warning is displayed once per session. 
This: command not found
bdcallen@igow-z640:/home/shared/uploads$ 2: `lang()` is deprecated as of rlang 0.2.0.
bash: command substitution: line 2: syntax error: unexpected end of file
2:: command not found
bdcallen@igow-z640:/home/shared/uploads$ Please use `call2()` instead.
bash: command substitution: line 2: syntax error: unexpected end of file
Please: command not found
bdcallen@igow-z640:/home/shared/uploads$ This warning is displayed once per session. 
This: command not found
bdcallen@igow-z640:/home/shared/uploads$ Importing speaker_data
Importing: command not found
bdcallen@igow-z640:/home/shared/uploads$ Importing speaker data.
Importing: command not found
bdcallen@igow-z640:/home/shared/uploads$    user  system elapsed 

Command 'user' not found, did you mean:

  command 'iuser' from deb ipmiutil
  command 'users' from deb coreutils
  command 'userv' from deb userv
  command 'fuser' from deb psmisc

Try: sudo apt install <deb name>

bdcallen@igow-z640:/home/shared/uploads$   0.215   0.008  59.453 
0.215: command not found
bdcallen@igow-z640:/home/shared/uploads$
bdcallen commented 4 years ago 
crsp=> select relname, description from pg_description
join pg_class on pg_description.objoid = pg_class.oid
where relname IN (SELECT table_name FROM information_schema.tables 
WHERE table_schema = 'streetevents');
        relname        |                                        description                                         
-----------------------+--------------------------------------------------------------------------------------------
 company_ids           | CREATED USING create_company_ids.R from GitHub iangow/se_core ON 2020-01-08 20:21:13
 calls_raw             | UPDATED USING import_calls_raw.R from GitHub iangow/se_core ON 2020-01-08 09:21:08
 calls_test            | CREATED USING create_calls_test.R from GitHub iangow/se_core ON 2019-12-06 17:12:29
 crsp_link             | CREATED USING create_crsp_link.R ON 2020-01-08 10:35:32
 manual_permno_matches | CREATED USING import_manual_permno_matches.R ON 2020-01-08 10:25:16 ON 2020-01-08 10:25:16
 calls                 | CREATED USING create_calls.R from GitHub iangow/se_core ON 2020-01-08 03:19:46
 selected_calls        | CREATED USING iangow/se_core/create_selected_calls.R ON 2020-01-08 20:21:28
(7 rows)
bdcallen commented 4 years ago

@iangow Not sure if script worked when I ran it, so checking tables and other outputs, and posting them here, in case you have an idea.

iangow commented 4 years ago

@iangow Tried running update_se.sh just now and got

bdcallen@igow-z640:~/se_core$ ./update_se.sh
Importing call_files
10.101.13.99
Updating data on 10.101.13.99 
Importing calls
Loading required package: DBI
Updating data on 10.101.13.99 
Warning messages:
1: `lang_name()` is deprecated as of rlang 0.2.0.
Please use `call_name()` instead.
This warning is displayed once per session. 
2: `lang()` is deprecated as of rlang 0.2.0.
Please use `call2()` instead.
This warning is displayed once per session. 
[1] 1318085
Error in (function (classes, fdef, mtable)  : 
  unable to find an inherited method for function ‘dbExecute’ for signature ‘"character", "missing"’
Calls: dbExecute -> <Anonymous>
Execution halted
Warning messages:
1: `lang_name()` is deprecated as of rlang 0.2.0.
Please use `call_name()` instead.
This warning is displayed once per session. 
2: `lang()` is deprecated as of rlang 0.2.0.
Please use `call2()` instead.
This warning is displayed once per session. 
Importing speaker_data
Importing speaker data.
   user  system elapsed 
  0.215   0.008  59.453

You have an error here:

Error in (function (classes, fdef, mtable) : unable to find an inherited method for function ‘dbExecute’ for signature ‘"character", "missing"’

Do you have the latest code? Let me see if I get the same error.

iangow commented 4 years ago

Seems that the code works for me:

igow@igow-z640:~/git/se_core$ git pull --rebase=preserve
Already up to date.
Successfully rebased and updated refs/heads/master.
igow@igow-z640:~/git/se_core$ ./update_se.sh
Importing call_files
localhost
Updating data on localhost
Joining, by = "file_path"
Importing calls
Loading required package: DBI
Updating data on localhost
Warning messages:
1: `lang_name()` is deprecated as of rlang 0.2.0.
Please use `call_name()` instead.
This warning is displayed once per session.
2: `lang()` is deprecated as of rlang 0.2.0.
Please use `call2()` instead.
This warning is displayed once per session.
[1] 1318281
Warning messages:
1: `lang_name()` is deprecated as of rlang 0.2.0.
Please use `call_name()` instead.
This warning is displayed once per session.
2: `lang()` is deprecated as of rlang 0.2.0.
Please use `call2()` instead.
This warning is displayed once per session.
Warning messages:
1: `lang_name()` is deprecated as of rlang 0.2.0.
Please use `call_name()` instead.
This warning is displayed once per session.
2: `lang()` is deprecated as of rlang 0.2.0.
Please use `call2()` instead.
This warning is displayed once per session.
Importing speaker_data
Importing speaker data.
[1] 196
[1] "Speaker data has 9750 rows"
[1] "Writing data to Postgres"
[1] "Writing dupe data to Postgres"
New rows:  TRUE
[1] 1
   user  system elapsed
  4.247   0.619 119.930
igow@igow-z640:~/git/se_core$
bdcallen commented 4 years ago

@iangow I assume I do have the latest code, as I git cloned the repository not long before Christmas. So perhaps the issue is with the permissions with wrds.

bdcallen commented 4 years ago

@iangow by the way, I have just cc'd you an email that I sent for a request to get a wrds account.