search

ome / omero-dropbox

OMERO.dropbox server for watching directories
https://www.openmicroscopy.org/omero
GNU General Public License v2.0
1 stars 6 forks source link

fsDropBoxMonitorClient: fix file cleanup (see #14) #15

Closed joshmoore closed 4 years ago

joshmoore commented 4 years ago

If login for a user failed, the importFile method failed since the finally blocked attempt to remove temporary files that had not yet been created.

joshmoore commented 4 years ago

fixes #12 - see https://travis-ci.org/github/ome/omero-dropbox/builds/669588582

++ omero hql -q -s root@localhost -w omero 'SELECT COUNT (*) FROM Image WHERE name like '\''%fake'\''' --style plain
1203
1204Found image: 0,4
1205+ result=0,4
1206+ '[' 0,4 = 0,4 ']'
1207+ echo
1208+ echo 'Found image: 0,4'

Will likely try to add a more complex scenario to try to capture other issues. cc: @will-moore

jburel commented 4 years ago

maybe an opportunity to improve the opening of t and to using with open(str(t) ...

 if t.exists():
     f = open(str(t), "r")
     lines = f.readlines()
     f.close()
joshmoore commented 4 years ago

I'd really rather get this out ASAP for testing by Fujimoto-san.

jburel commented 4 years ago

in that case, changes look sensible

joshmoore commented 4 years ago

Cheers.

pwalczysko commented 4 years ago

@joshmoore Unfortunately, still an error occurs after the start of the OMERO.server in DropBox.log, see below for the whole sequence of commands I ran on my Mac.

(env561) pwalczysko@ls31618~$ pip uninstall omero-dropbox
(env561) pwalczysko@ls31618~$ pip install -e git://github.com/ome/omero-dropbox.git@master#egg=omero-dropbox
Obtaining omero-dropbox from git+git://github.com/ome/omero-dropbox.git@master#egg=omero-dropbox
  WARNING: git clone in ./src/omero-dropbox exists with URL git://github.com/joshmoore/omero-dropbox.git
  WARNING: The plan is to install the git repository git://github.com/ome/omero-dropbox.git
What to do?  (s)witch, (i)gnore, (w)ipe, (b)ackup w
  WARNING: Deleting ./src/omero-dropbox
  Cloning git://github.com/ome/omero-dropbox.git (to revision master) to ./src/omero-dropbox
  Running command git clone -q git://github.com/ome/omero-dropbox.git /Users/pwalczysko/src/omero-dropbox
Requirement already satisfied: omero-py in ./miniconda3/envs/env561/lib/python3.7/site-packages (from omero-dropbox) (5.6.2)
Requirement already satisfied: Pillow in ./miniconda3/envs/env561/lib/python3.7/site-packages (from omero-py->omero-dropbox) (7.0.0)
Requirement already satisfied: numpy in ./miniconda3/envs/env561/lib/python3.7/site-packages (from omero-py->omero-dropbox) (1.18.1)
Requirement already satisfied: future in ./miniconda3/envs/env561/lib/python3.7/site-packages (from omero-py->omero-dropbox) (0.18.2)
Requirement already satisfied: zeroc-ice<3.7,>=3.6.4 in ./miniconda3/envs/env561/lib/python3.7/site-packages (from omero-py->omero-dropbox) (3.6.5)
Installing collected packages: omero-dropbox
  Running setup.py develop for omero-dropbox
Successfully installed omero-dropbox
(env561) pwalczysko@ls31618~$ omero admin start
No descriptor given. Using etc/grid/default.xml
Waiting on startup. Use CTRL-C to exit
(env561) pwalczysko@ls31618~$ omero admin diagnostics

================================================================================
OMERO Diagnostics (admin) 5.6.2
================================================================================

Commands:   java -version                  1.8.0     (/usr/bin/java)
Commands:   python -V                      3.7.6     (/Users/pwalczysko/miniconda3/envs/env561/bin/python -- 3 others)
Commands:   icegridnode --version          3.6.5     (/usr/local/bin/icegridnode)
Commands:   icegridadmin --version         3.6.5     (/usr/local/bin/icegridadmin)
Commands:   psql --version                 12.2      (/usr/local/bin/psql)
Commands:   openssl version                1.1.1     (/Users/pwalczysko/miniconda3/envs/env561/bin/openssl -- 2 others)

Server:     icegridnode                    running
Server:     Blitz-0                        active (pid = 94383, enabled)
Server:     DropBox                        active (pid = 94385, enabled)
Server:     FileServer                     active (pid = 94386, enabled)
Server:     Indexer-0                      activating (enabled)
Server:     MonitorServer                  active (pid = 94388, enabled)
Server:     OMERO.Glacier2                 active (pid = 94389, enabled)
Server:     OMERO.IceStorm                 active (pid = 94390, enabled)
Server:     PixelData-0                    activating (enabled)
Server:     Processor-0                    active (pid = 94392, enabled)
Server:     Tables-0                       inactive (disabled)
Server:     TestDropBox                    inactive (enabled)

Log dir:    /Users/pwalczysko/Downloads/OMERO.server-5.6.1-ice36-b225/var/log exists
Log files:  Blitz-0.log                    6.8 MB        errors=0    warnings=29  
Log files:  DropBox.log                    9.5 KB        errors=3    warnings=8   
Log files:  FileServer.log                 1.2 KB        errors=0    warnings=4   
Log files:  Indexer-0.log                  25.6 KB       errors=0    warnings=12  
Log files:  MonitorServer.log              4.1 KB        errors=3    warnings=4   
Log files:  OMEROweb.lock                  empty
Log files:  OMEROweb.log                   39.2 KB       errors=0    warnings=1   
Log files:  PixelData-0.log                19.3 KB       errors=0    warnings=12  
Log files:  Processor-0.log                395.9 KB      errors=0    warnings=15  
Log files:  Tables-0.log                   n/a
Log files:  TestDropBox.log                n/a
Log files:  master.err                     271.1 KB      errors=2    warnings=2   
Log files:  master.out                     empty
Log files:  Total size                     7.54 MB

Environment:OMERO_HOME=(unset)             
Environment:OMERODIR=/Users/pwalczysko/Downloads/OMERO.server-5.6.1-ice36-b225 
Environment:OMERO_NODE=(unset)             
Environment:OMERO_MASTER=(unset)           
Environment:OMERO_USERDIR=(unset)          
Environment:OMERO_TMPDIR=(unset)           
Environment:PATH=/Users/pwalczysko/miniconda3/envs/env561/bin:/Users/pwalczysko/miniconda3/condabin:/Users/pwalczysko/Applications/Aspera CLI/bin:/usr/local/opt/ruby/bin:/Users/pwalczysko/.gem/ruby/2.6.0/bin:/usr/local/opt/sphinx-doc/bin:/usr/local/opt/python2/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin 
Environment:PYTHONPATH=(unset)             
Environment:ICE_HOME=(unset)               
Environment:LD_LIBRARY_PATH=(unset)        
Environment:DYLD_LIBRARY_PATH=(unset)      

OMERO SSL port:4064                           
OMERO TCP port:4063                           
OMERO data dir:'/Users/pwalczysko/var/561/'   Exists? True  Is writable? True
OMERO temp dir:'/Users/pwalczysko/omero/tmp'  Exists? True  Is writable? True   (Size: 0)

JVM settings: Blitz-${index}                -Xmx2576m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Indexer-${index}              -Xmx1717m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: PixelData-${index}            -Xmx2576m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Repository-${index}           -Xmx1717m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
(env561) pwalczysko@ls31618~$ 

(env561) pwalczysko@ls31618~$ cat /Users/pwalczysko/Downloads/OMERO.server-5.6.1-ice36-b225/var/log/DropBox.log 

2020-04-02 19:14:26,697 INFO  [                             omero.utils] (MainThread) Failed to get session on attempt 1
2020-04-02 19:14:29,838 INFO  [                        fsclient.DropBox] (MainThread) Monitor parameters = {'default': {'watchDir': '/Users/pwalczysko/var/omero/DropBox/', 'eventTypes': [Creation, Modification], 'pathMode': Follow, 'whitelist': [], 'blacklist': [], 'timeout': 0.0, 'blockSize': 0, 'ignoreSysFiles': True, 'ignoreDirEvents': True, 'dirImportWait': 60, 'fileWait': 15.0, 'throttleImport': 10, 'timeToLive': 0, 'timeToIdle': 600000, 'fileBatch': 10, 'readers': '', 'importArgs': ''}}
2020-04-02 19:14:29,857 INFO  [                        fsclient.DropBox] (MainThread) Creating client for user: default
2020-04-02 19:14:29,859 INFO  [                    omero.util.Resources] (Thread-2  ) Starting
2020-04-02 19:14:29,861 INFO  [         fsclient.fsDropBoxMonitorClient] (MainThread) EVENT_RECORD::Cookie::1585851269.860974::Directory::/Users/pwalczysko/var/omero/DropBox/
2020-04-02 19:14:29,891 ERROR [                        fsclient.DropBox] (MainThread) Failed create or start monitor : 
Traceback (most recent call last):
  File "/Users/pwalczysko/src/omero-dropbox/src/fsDropBox.py", line 209, in run
    mClientProxy)
  File "/Users/pwalczysko/miniconda3/envs/env561/lib/python3.7/site-packages/omero_FS_ice.py", line 1469, in createMonitor
    return _M_omero.grid.monitors.MonitorServer._op_createMonitor.invoke(self, ((mType, eTypes, pMode, pathString, whitelist, blacklist, timeout, blockSize, ignoreSysFiles, ignoreDirEvents, platformCheck, proxy), _ctx))
omero.OmeroFSError: exception ::omero::OmeroFSError
{
    serverStackTrace = 
    serverExceptionClass = 
    message = 
    reason = Failed to create monitor: No module named 'Foundation'
}
2020-04-02 19:14:29,894 INFO  [                        fsclient.DropBox] (MainThread) Started OMERO.fs DropBox client
(env561) pwalczysko@ls31618~$
joshmoore commented 4 years ago

There are a couple of different ways for tackling that. I think @jburel had a suggested one.

pwalczysko commented 4 years ago

@joshmoore Unfortunately, not aware of @jburel suggesting a solution. Maybe you mean https://trello.com/c/lLaWDumZ/23-macosx-server#comment-5e7b562817de3d1ae0381bb4 - but this is just report of "works for me" ? @jburel ?

jburel commented 4 years ago

@pwalczysko try to run in the conda env pip install pyobjc we discussed in general, this might have been while you were off

pwalczysko commented 4 years ago

@jburel Thank you. Installed the pyobjc. After that, no error in DrobBox.log after server restart.

But, the DropBox does not seem to work. Do not see any errors in the logs, but no activity either, after I dropped couple of screenshots into the user-2 and user-3 folders in /Users/pwalczysko/var/omero/DropBox/user-x location

cc @joshmoore

From the DropBox.log. 

2020-04-03 09:24:26,605 INFO  [                             omero.utils] (MainThread) Failed to get session on attempt 1
2020-04-03 09:24:30,486 INFO  [                        fsclient.DropBox] (MainThread) Monitor parameters = {'default': {'watchDir': '/Users/pwalczysko/var/omero/DropBox/', 'eventTypes': [Creation, Modification], 'pathMode': Follow, 'whitelist': [], 'blacklist': [], 'timeout': 0.0, 'blockSize': 0, 'ignoreSysFiles': True, 'ignoreDirEvents': True, 'dirImportWait': 60, 'fileWait': 15.0, 'throttleImport': 10, 'timeToLive': 0, 'timeToIdle': 600000, 'fileBatch': 10, 'readers': '', 'importArgs': ''}}
2020-04-03 09:24:30,515 INFO  [                        fsclient.DropBox] (MainThread) Creating client for user: default
2020-04-03 09:24:30,517 INFO  [                    omero.util.Resources] (Thread-2  ) Starting
2020-04-03 09:24:30,519 INFO  [         fsclient.fsDropBoxMonitorClient] (MainThread) EVENT_RECORD::Cookie::1585902270.519021::Directory::/Users/pwalczysko/var/omero/DropBox/
2020-04-03 09:24:30,975 INFO  [                        fsclient.DropBox] (MainThread) Created monitor with id = 8a556430-7584-11ea-bbc9-acbc32a4abb3
2020-04-03 09:24:30,980 INFO  [                        fsclient.DropBox] (MainThread) Started OMERO.fs DropBox client
^C
(env561) pwalczysko@ls31618~$ tail -f  /Users/pwalczysko/Downloads/OMERO.server-5.6.1-ice36-b225/var/log/DropBox.log 
2020-04-02 19:24:14,528 WARNI [                          StreamRedirect] (MainThread) No attribute: closed
2020-04-02 19:24:14,560 INFO  [         fsclient.fsDropBoxMonitorClient] (MainThread) Stopping state...
2020-04-02 19:24:14,569 INFO  [         fsclient.fsDropBoxMonitorClient] (MainThread) Cleaning up resources state...
2020-04-03 09:24:26,605 INFO  [                             omero.utils] (MainThread) Failed to get session on attempt 1
2020-04-03 09:24:30,486 INFO  [                        fsclient.DropBox] (MainThread) Monitor parameters = {'default': {'watchDir': '/Users/pwalczysko/var/omero/DropBox/', 'eventTypes': [Creation, Modification], 'pathMode': Follow, 'whitelist': [], 'blacklist': [], 'timeout': 0.0, 'blockSize': 0, 'ignoreSysFiles': True, 'ignoreDirEvents': True, 'dirImportWait': 60, 'fileWait': 15.0, 'throttleImport': 10, 'timeToLive': 0, 'timeToIdle': 600000, 'fileBatch': 10, 'readers': '', 'importArgs': ''}}
2020-04-03 09:24:30,515 INFO  [                        fsclient.DropBox] (MainThread) Creating client for user: default
2020-04-03 09:24:30,517 INFO  [                    omero.util.Resources] (Thread-2  ) Starting
2020-04-03 09:24:30,519 INFO  [         fsclient.fsDropBoxMonitorClient] (MainThread) EVENT_RECORD::Cookie::1585902270.519021::Directory::/Users/pwalczysko/var/omero/DropBox/
2020-04-03 09:24:30,975 INFO  [                        fsclient.DropBox] (MainThread) Created monitor with id = 8a556430-7584-11ea-bbc9-acbc32a4abb3
2020-04-03 09:24:30,980 INFO  [                        fsclient.DropBox] (MainThread) Started OMERO.fs DropBox client
will-moore commented 4 years ago

@pwalczysko What's in the MonitorServer.log ? That's where I saw an error.

pwalczysko commented 4 years ago

@will-moore thanks. Indeed, in MonitorServer.log there is an error:

2020-04-03 09:24:16,494 INFO  [                  fsserver.MonitorServer] (MainThread) Started OMERO.fs MonitorServer
2020-04-03 09:24:30,969 INFO  [             fsserver.fsMac-10-5-Monitor] (Dummy-3   ) Monitor set-up on /Users/pwalczysko/var/omero/DropBox/
2020-04-03 09:24:30,970 INFO  [             fsserver.fsMac-10-5-Monitor] (Dummy-3   ) Monitoring ['Creation', 'Modification'] events
2020-04-03 09:24:30,974 INFO  [        fsclient.fsNotificationScheduler] (Thread-4  ) Notification Scheduler running
2020-04-03 09:24:30,975 INFO  [                fsserver.fsMonitorServer] (Dummy-3   ) Monitor id = 8a556430-7584-11ea-bbc9-acbc32a4abb3 created. Proxy: DropBox.default -t -e 1.1 @ DropBox.omerofs.DropBox
2020-04-03 09:24:30,978 INFO  [                fsserver.fsMonitorServer] (Dummy-5   ) Monitor id = 8a556430-7584-11ea-bbc9-acbc32a4abb3 started
2020-04-03 09:58:32,531 WARNI [                                  stderr] (Thread-2  ) Exception in thread Thread-2:
Traceback (most recent call last):
  File "/Users/pwalczysko/miniconda3/envs/env561/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/Users/pwalczysko/src/omero-dropbox/src/fsMac-10-5-Monitor.py", line 157, in run
    FSEvents.CFRunLoopRun()
  File "/Users/pwalczysko/src/omero-dropbox/src/fsMac-10-5-Monitor.py", line 217, in callback
    new, old, chg = dir.getChangedFiles(eventPaths[i])
  File "/Users/pwalczysko/src/omero-dropbox/src/fsDirectory.py", line 380, in getChangedFiles
    newTree, oldTree = self.patchTree(pathString)
  File "/Users/pwalczysko/src/omero-dropbox/src/fsDirectory.py", line 216, in patchTree
    newTree = DirNode(path, self, self.pathMode)
  File "/Users/pwalczysko/src/omero-dropbox/src/fsDirectory.py", line 547, in __init__
    Node.__init__(self, path)
  File "/Users/pwalczysko/src/omero-dropbox/src/fsDirectory.py", line 422, in __init__
    self.owner = path.owner
  File "/Users/pwalczysko/miniconda3/envs/env561/lib/python3.7/site-packages/omero_ext/path.py", line 1169, in __get_owner_unix
    st = self.stat()
  File "/Users/pwalczysko/miniconda3/envs/env561/lib/python3.7/site-packages/omero_ext/path.py", line 1138, in stat
    return os.stat(self)
FileNotFoundError: [Errno 2] No such file or directory: path("b'/Users/pwalczysko/var/omero/DropBox/user-2/'")
pwalczysko commented 4 years ago

But that dir does exist Do not get where does DropBox get the strange b in front of the path, but otherwise see below

(env561) pwalczysko@ls31618~/Downloads/OMERO.server-5.6.1-ice36-b225$ omero config get
omero.data.dir=/Users/pwalczysko/var/561/
omero.db.name=561
omero.db.pass=********
omero.db.user=omero
omero.fs.watchDir=/Users/pwalczysko/var/omero/DropBox/
omero.web.application_server=development
omero.web.debug=True
(env561) pwalczysko@ls31618~/Downloads/OMERO.server-5.6.1-ice36-b225$ cd /Users/pwalczysko/var/omero/DropBox/user-2/
(env561) pwalczysko@ls31618~/var/omero/DropBox/user-2$
pwalczysko commented 4 years ago

Deleted everything under the DropBox folder, then restarted the server again. After that, in the MonitorServer.log I get

...
FileNotFoundError: [Errno 2] No such file or directory: path("b'/Users/pwalczysko/var/omero/DropBox/'")

Why then ? What does "b" mean ?

jburel commented 4 years ago

it looks like issue due to py2->py3 migration

will-moore commented 4 years ago

That's the same error I'm seeing:

FileNotFoundError: [Errno 2] No such file or directory: path("b'/OMERO/DropBox/'")
pwalczysko commented 4 years ago

@will-moore yuhuu, I am not alone there...

jburel commented 4 years ago

Strange that i did not get the error when I did some tests. I can try to dig into it later on