Closed joshmoore closed 4 years ago
fixes #12 - see https://travis-ci.org/github/ome/omero-dropbox/builds/669588582
++ omero hql -q -s root@localhost -w omero 'SELECT COUNT (*) FROM Image WHERE name like '\''%fake'\''' --style plain
1203
1204Found image: 0,4
1205+ result=0,4
1206+ '[' 0,4 = 0,4 ']'
1207+ echo
1208+ echo 'Found image: 0,4'
Will likely try to add a more complex scenario to try to capture other issues. cc: @will-moore
maybe an opportunity to improve the opening of t
and to
using with open(str(t) ...
if t.exists():
f = open(str(t), "r")
lines = f.readlines()
f.close()
I'd really rather get this out ASAP for testing by Fujimoto-san.
in that case, changes look sensible
Cheers.
@joshmoore Unfortunately, still an error occurs after the start of the OMERO.server in DropBox.log, see below for the whole sequence of commands I ran on my Mac.
(env561) pwalczysko@ls31618~$ pip uninstall omero-dropbox
(env561) pwalczysko@ls31618~$ pip install -e git://github.com/ome/omero-dropbox.git@master#egg=omero-dropbox
Obtaining omero-dropbox from git+git://github.com/ome/omero-dropbox.git@master#egg=omero-dropbox
WARNING: git clone in ./src/omero-dropbox exists with URL git://github.com/joshmoore/omero-dropbox.git
WARNING: The plan is to install the git repository git://github.com/ome/omero-dropbox.git
What to do? (s)witch, (i)gnore, (w)ipe, (b)ackup w
WARNING: Deleting ./src/omero-dropbox
Cloning git://github.com/ome/omero-dropbox.git (to revision master) to ./src/omero-dropbox
Running command git clone -q git://github.com/ome/omero-dropbox.git /Users/pwalczysko/src/omero-dropbox
Requirement already satisfied: omero-py in ./miniconda3/envs/env561/lib/python3.7/site-packages (from omero-dropbox) (5.6.2)
Requirement already satisfied: Pillow in ./miniconda3/envs/env561/lib/python3.7/site-packages (from omero-py->omero-dropbox) (7.0.0)
Requirement already satisfied: numpy in ./miniconda3/envs/env561/lib/python3.7/site-packages (from omero-py->omero-dropbox) (1.18.1)
Requirement already satisfied: future in ./miniconda3/envs/env561/lib/python3.7/site-packages (from omero-py->omero-dropbox) (0.18.2)
Requirement already satisfied: zeroc-ice<3.7,>=3.6.4 in ./miniconda3/envs/env561/lib/python3.7/site-packages (from omero-py->omero-dropbox) (3.6.5)
Installing collected packages: omero-dropbox
Running setup.py develop for omero-dropbox
Successfully installed omero-dropbox
(env561) pwalczysko@ls31618~$ omero admin start
No descriptor given. Using etc/grid/default.xml
Waiting on startup. Use CTRL-C to exit
(env561) pwalczysko@ls31618~$ omero admin diagnostics
================================================================================
OMERO Diagnostics (admin) 5.6.2
================================================================================
Commands: java -version 1.8.0 (/usr/bin/java)
Commands: python -V 3.7.6 (/Users/pwalczysko/miniconda3/envs/env561/bin/python -- 3 others)
Commands: icegridnode --version 3.6.5 (/usr/local/bin/icegridnode)
Commands: icegridadmin --version 3.6.5 (/usr/local/bin/icegridadmin)
Commands: psql --version 12.2 (/usr/local/bin/psql)
Commands: openssl version 1.1.1 (/Users/pwalczysko/miniconda3/envs/env561/bin/openssl -- 2 others)
Server: icegridnode running
Server: Blitz-0 active (pid = 94383, enabled)
Server: DropBox active (pid = 94385, enabled)
Server: FileServer active (pid = 94386, enabled)
Server: Indexer-0 activating (enabled)
Server: MonitorServer active (pid = 94388, enabled)
Server: OMERO.Glacier2 active (pid = 94389, enabled)
Server: OMERO.IceStorm active (pid = 94390, enabled)
Server: PixelData-0 activating (enabled)
Server: Processor-0 active (pid = 94392, enabled)
Server: Tables-0 inactive (disabled)
Server: TestDropBox inactive (enabled)
Log dir: /Users/pwalczysko/Downloads/OMERO.server-5.6.1-ice36-b225/var/log exists
Log files: Blitz-0.log 6.8 MB errors=0 warnings=29
Log files: DropBox.log 9.5 KB errors=3 warnings=8
Log files: FileServer.log 1.2 KB errors=0 warnings=4
Log files: Indexer-0.log 25.6 KB errors=0 warnings=12
Log files: MonitorServer.log 4.1 KB errors=3 warnings=4
Log files: OMEROweb.lock empty
Log files: OMEROweb.log 39.2 KB errors=0 warnings=1
Log files: PixelData-0.log 19.3 KB errors=0 warnings=12
Log files: Processor-0.log 395.9 KB errors=0 warnings=15
Log files: Tables-0.log n/a
Log files: TestDropBox.log n/a
Log files: master.err 271.1 KB errors=2 warnings=2
Log files: master.out empty
Log files: Total size 7.54 MB
Environment:OMERO_HOME=(unset)
Environment:OMERODIR=/Users/pwalczysko/Downloads/OMERO.server-5.6.1-ice36-b225
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:OMERO_USERDIR=(unset)
Environment:OMERO_TMPDIR=(unset)
Environment:PATH=/Users/pwalczysko/miniconda3/envs/env561/bin:/Users/pwalczysko/miniconda3/condabin:/Users/pwalczysko/Applications/Aspera CLI/bin:/usr/local/opt/ruby/bin:/Users/pwalczysko/.gem/ruby/2.6.0/bin:/usr/local/opt/sphinx-doc/bin:/usr/local/opt/python2/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
Environment:PYTHONPATH=(unset)
Environment:ICE_HOME=(unset)
Environment:LD_LIBRARY_PATH=(unset)
Environment:DYLD_LIBRARY_PATH=(unset)
OMERO SSL port:4064
OMERO TCP port:4063
OMERO data dir:'/Users/pwalczysko/var/561/' Exists? True Is writable? True
OMERO temp dir:'/Users/pwalczysko/omero/tmp' Exists? True Is writable? True (Size: 0)
JVM settings: Blitz-${index} -Xmx2576m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Indexer-${index} -Xmx1717m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: PixelData-${index} -Xmx2576m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Repository-${index} -Xmx1717m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
(env561) pwalczysko@ls31618~$
(env561) pwalczysko@ls31618~$ cat /Users/pwalczysko/Downloads/OMERO.server-5.6.1-ice36-b225/var/log/DropBox.log
2020-04-02 19:14:26,697 INFO [ omero.utils] (MainThread) Failed to get session on attempt 1
2020-04-02 19:14:29,838 INFO [ fsclient.DropBox] (MainThread) Monitor parameters = {'default': {'watchDir': '/Users/pwalczysko/var/omero/DropBox/', 'eventTypes': [Creation, Modification], 'pathMode': Follow, 'whitelist': [], 'blacklist': [], 'timeout': 0.0, 'blockSize': 0, 'ignoreSysFiles': True, 'ignoreDirEvents': True, 'dirImportWait': 60, 'fileWait': 15.0, 'throttleImport': 10, 'timeToLive': 0, 'timeToIdle': 600000, 'fileBatch': 10, 'readers': '', 'importArgs': ''}}
2020-04-02 19:14:29,857 INFO [ fsclient.DropBox] (MainThread) Creating client for user: default
2020-04-02 19:14:29,859 INFO [ omero.util.Resources] (Thread-2 ) Starting
2020-04-02 19:14:29,861 INFO [ fsclient.fsDropBoxMonitorClient] (MainThread) EVENT_RECORD::Cookie::1585851269.860974::Directory::/Users/pwalczysko/var/omero/DropBox/
2020-04-02 19:14:29,891 ERROR [ fsclient.DropBox] (MainThread) Failed create or start monitor :
Traceback (most recent call last):
File "/Users/pwalczysko/src/omero-dropbox/src/fsDropBox.py", line 209, in run
mClientProxy)
File "/Users/pwalczysko/miniconda3/envs/env561/lib/python3.7/site-packages/omero_FS_ice.py", line 1469, in createMonitor
return _M_omero.grid.monitors.MonitorServer._op_createMonitor.invoke(self, ((mType, eTypes, pMode, pathString, whitelist, blacklist, timeout, blockSize, ignoreSysFiles, ignoreDirEvents, platformCheck, proxy), _ctx))
omero.OmeroFSError: exception ::omero::OmeroFSError
{
serverStackTrace =
serverExceptionClass =
message =
reason = Failed to create monitor: No module named 'Foundation'
}
2020-04-02 19:14:29,894 INFO [ fsclient.DropBox] (MainThread) Started OMERO.fs DropBox client
(env561) pwalczysko@ls31618~$
There are a couple of different ways for tackling that. I think @jburel had a suggested one.
@joshmoore Unfortunately, not aware of @jburel suggesting a solution. Maybe you mean https://trello.com/c/lLaWDumZ/23-macosx-server#comment-5e7b562817de3d1ae0381bb4 - but this is just report of "works for me" ? @jburel ?
@pwalczysko
try to run in the conda env pip install pyobjc
we discussed in general, this might have been while you were off
@jburel Thank you. Installed the pyobjc. After that, no error in DrobBox.log after server restart.
But, the DropBox does not seem to work. Do not see any errors in the logs, but no activity either, after I dropped couple of screenshots into the user-2
and user-3
folders in /Users/pwalczysko/var/omero/DropBox/user-x
location
cc @joshmoore
From the DropBox.log.
2020-04-03 09:24:26,605 INFO [ omero.utils] (MainThread) Failed to get session on attempt 1
2020-04-03 09:24:30,486 INFO [ fsclient.DropBox] (MainThread) Monitor parameters = {'default': {'watchDir': '/Users/pwalczysko/var/omero/DropBox/', 'eventTypes': [Creation, Modification], 'pathMode': Follow, 'whitelist': [], 'blacklist': [], 'timeout': 0.0, 'blockSize': 0, 'ignoreSysFiles': True, 'ignoreDirEvents': True, 'dirImportWait': 60, 'fileWait': 15.0, 'throttleImport': 10, 'timeToLive': 0, 'timeToIdle': 600000, 'fileBatch': 10, 'readers': '', 'importArgs': ''}}
2020-04-03 09:24:30,515 INFO [ fsclient.DropBox] (MainThread) Creating client for user: default
2020-04-03 09:24:30,517 INFO [ omero.util.Resources] (Thread-2 ) Starting
2020-04-03 09:24:30,519 INFO [ fsclient.fsDropBoxMonitorClient] (MainThread) EVENT_RECORD::Cookie::1585902270.519021::Directory::/Users/pwalczysko/var/omero/DropBox/
2020-04-03 09:24:30,975 INFO [ fsclient.DropBox] (MainThread) Created monitor with id = 8a556430-7584-11ea-bbc9-acbc32a4abb3
2020-04-03 09:24:30,980 INFO [ fsclient.DropBox] (MainThread) Started OMERO.fs DropBox client
^C
(env561) pwalczysko@ls31618~$ tail -f /Users/pwalczysko/Downloads/OMERO.server-5.6.1-ice36-b225/var/log/DropBox.log
2020-04-02 19:24:14,528 WARNI [ StreamRedirect] (MainThread) No attribute: closed
2020-04-02 19:24:14,560 INFO [ fsclient.fsDropBoxMonitorClient] (MainThread) Stopping state...
2020-04-02 19:24:14,569 INFO [ fsclient.fsDropBoxMonitorClient] (MainThread) Cleaning up resources state...
2020-04-03 09:24:26,605 INFO [ omero.utils] (MainThread) Failed to get session on attempt 1
2020-04-03 09:24:30,486 INFO [ fsclient.DropBox] (MainThread) Monitor parameters = {'default': {'watchDir': '/Users/pwalczysko/var/omero/DropBox/', 'eventTypes': [Creation, Modification], 'pathMode': Follow, 'whitelist': [], 'blacklist': [], 'timeout': 0.0, 'blockSize': 0, 'ignoreSysFiles': True, 'ignoreDirEvents': True, 'dirImportWait': 60, 'fileWait': 15.0, 'throttleImport': 10, 'timeToLive': 0, 'timeToIdle': 600000, 'fileBatch': 10, 'readers': '', 'importArgs': ''}}
2020-04-03 09:24:30,515 INFO [ fsclient.DropBox] (MainThread) Creating client for user: default
2020-04-03 09:24:30,517 INFO [ omero.util.Resources] (Thread-2 ) Starting
2020-04-03 09:24:30,519 INFO [ fsclient.fsDropBoxMonitorClient] (MainThread) EVENT_RECORD::Cookie::1585902270.519021::Directory::/Users/pwalczysko/var/omero/DropBox/
2020-04-03 09:24:30,975 INFO [ fsclient.DropBox] (MainThread) Created monitor with id = 8a556430-7584-11ea-bbc9-acbc32a4abb3
2020-04-03 09:24:30,980 INFO [ fsclient.DropBox] (MainThread) Started OMERO.fs DropBox client
@pwalczysko What's in the MonitorServer.log
?
That's where I saw an error.
@will-moore thanks. Indeed, in MonitorServer.log
there is an error:
2020-04-03 09:24:16,494 INFO [ fsserver.MonitorServer] (MainThread) Started OMERO.fs MonitorServer
2020-04-03 09:24:30,969 INFO [ fsserver.fsMac-10-5-Monitor] (Dummy-3 ) Monitor set-up on /Users/pwalczysko/var/omero/DropBox/
2020-04-03 09:24:30,970 INFO [ fsserver.fsMac-10-5-Monitor] (Dummy-3 ) Monitoring ['Creation', 'Modification'] events
2020-04-03 09:24:30,974 INFO [ fsclient.fsNotificationScheduler] (Thread-4 ) Notification Scheduler running
2020-04-03 09:24:30,975 INFO [ fsserver.fsMonitorServer] (Dummy-3 ) Monitor id = 8a556430-7584-11ea-bbc9-acbc32a4abb3 created. Proxy: DropBox.default -t -e 1.1 @ DropBox.omerofs.DropBox
2020-04-03 09:24:30,978 INFO [ fsserver.fsMonitorServer] (Dummy-5 ) Monitor id = 8a556430-7584-11ea-bbc9-acbc32a4abb3 started
2020-04-03 09:58:32,531 WARNI [ stderr] (Thread-2 ) Exception in thread Thread-2:
Traceback (most recent call last):
File "/Users/pwalczysko/miniconda3/envs/env561/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/Users/pwalczysko/src/omero-dropbox/src/fsMac-10-5-Monitor.py", line 157, in run
FSEvents.CFRunLoopRun()
File "/Users/pwalczysko/src/omero-dropbox/src/fsMac-10-5-Monitor.py", line 217, in callback
new, old, chg = dir.getChangedFiles(eventPaths[i])
File "/Users/pwalczysko/src/omero-dropbox/src/fsDirectory.py", line 380, in getChangedFiles
newTree, oldTree = self.patchTree(pathString)
File "/Users/pwalczysko/src/omero-dropbox/src/fsDirectory.py", line 216, in patchTree
newTree = DirNode(path, self, self.pathMode)
File "/Users/pwalczysko/src/omero-dropbox/src/fsDirectory.py", line 547, in __init__
Node.__init__(self, path)
File "/Users/pwalczysko/src/omero-dropbox/src/fsDirectory.py", line 422, in __init__
self.owner = path.owner
File "/Users/pwalczysko/miniconda3/envs/env561/lib/python3.7/site-packages/omero_ext/path.py", line 1169, in __get_owner_unix
st = self.stat()
File "/Users/pwalczysko/miniconda3/envs/env561/lib/python3.7/site-packages/omero_ext/path.py", line 1138, in stat
return os.stat(self)
FileNotFoundError: [Errno 2] No such file or directory: path("b'/Users/pwalczysko/var/omero/DropBox/user-2/'")
But that dir does exist
Do not get where does DropBox get the strange b
in front of the path, but otherwise see below
(env561) pwalczysko@ls31618~/Downloads/OMERO.server-5.6.1-ice36-b225$ omero config get
omero.data.dir=/Users/pwalczysko/var/561/
omero.db.name=561
omero.db.pass=********
omero.db.user=omero
omero.fs.watchDir=/Users/pwalczysko/var/omero/DropBox/
omero.web.application_server=development
omero.web.debug=True
(env561) pwalczysko@ls31618~/Downloads/OMERO.server-5.6.1-ice36-b225$ cd /Users/pwalczysko/var/omero/DropBox/user-2/
(env561) pwalczysko@ls31618~/var/omero/DropBox/user-2$
Deleted everything under the DropBox folder, then restarted the server again.
After that, in the MonitorServer.log
I get
...
FileNotFoundError: [Errno 2] No such file or directory: path("b'/Users/pwalczysko/var/omero/DropBox/'")
Why then ? What does "b" mean ?
it looks like issue due to py2->py3 migration
That's the same error I'm seeing:
FileNotFoundError: [Errno 2] No such file or directory: path("b'/OMERO/DropBox/'")
@will-moore yuhuu, I am not alone there...
Strange that i did not get the error when I did some tests. I can try to dig into it later on
If login for a user failed, the importFile method failed since the finally blocked attempt to remove temporary files that had not yet been created.