search

iangow / se_core

Core code for StreetEvents data
7 stars 5 forks source link

Investigate permissions issues #11

Closed iangow closed 3 years ago

iangow commented 4 years ago

@bdcallen I'm still seeing permissions issues (see below). I can fix these with sudo chown igow:mccgr -R uploads run from /home/shared/, but I'm not sure what is causing this. With the wrong ownership, get_xml_files.sh does not run successfully. It's possible that jpk and mschabus share the same numeric IDs as the users who own the files on the original server that get_xml_files.sh gets the files from. When I run get_xml_files.sh, the files are owned by igow:igow (i.e., me). But things may work differently with the cron job, which may be run by root.

igow@igow-z640:~$ cd $SE_DIR
igow@igow-z640:/home/shared/uploads$ ls -hlt
total 488M
-rw-r--r-- 1 jpk mschabus  41K Jan  8 00:00 12953641_T.xml
-rw-r--r-- 1 jpk mschabus  31K Jan  7 23:54 12446155_T.xml
-rw-r--r-- 1 jpk mschabus 343K Jan  7 23:50 12896966_T.xml
-rw-r--r-- 1 jpk mschabus  72K Jan  7 23:43 12947910_T.xml
-rw-r--r-- 1 jpk mschabus  38K Jan  7 23:41 12952857_T.xml
-rw-r--r-- 1 jpk mschabus 223K Jan  7 23:38 12883625_T.xml
-rw-r--r-- 1 jpk mschabus  66K Jan  7 23:32 12951700_T.xml
-rw-r--r-- 1 jpk mschabus  92K Jan  7 23:27 12937134_T.xml
-rw-r--r-- 1 jpk mschabus 318K Jan  7 23:23 12771277_T.xml
-rw-r--r-- 1 jpk mschabus 109K Jan  7 23:04 12281077_T.xml
iangow commented 4 years ago

@bdcallen Could you please do some research on this issue, which persists? It seems that the CRON job may not be affected by this issue, but I guess is what is causing it in the first place. It may be useful to check the underlying numerical IDs for jpk and mschabus on the Z640 box and compare that with the IDs associated with the owner and group of the files on the server we're getting the XML files from.

iangow commented 3 years ago

On the Research Platforms virtual machine:

iangow@sftp-resplat:/home/thomsonreuters$ ls -hlt
total 15G
drwxr-xr-x 4 thomsonreuters thomsonreuters  15M Nov 16 14:05 uploads
-rw-r--r-- 1 root           root           4.5G Sep 28 17:30 2020-09-28-uploads.tar.xz
drwxr-xr-x 2 thomsonreuters root           4.0K Sep 28 11:14 bin
-rw------- 1 thomsonreuters root             44 Sep 28 05:56 ftp-setranscripts.refinitiv.com.auth
-rw-r--r-- 1 root           root           4.5G Sep 21 17:26 2020-09-21-uploads.tar.xz
-rw-r--r-- 1 thomsonreuters sftpusers      6.0G Oct  4  2019 backup_20191004.tar.g
iangow@sftp-resplat:/home/thomsonreuters$ id -u thomsonreuters
1001
iangow@sftp-resplat:/home/thomsonreuters$ id -G thomsonreuters
1002

On Z640:

igow@igow-z640:/home/shared$ id -u jpk
1001
igow@igow-z640:/home/shared$ id -u mschabus
1002

I think I will try the --chown option in rsync.