Open atanasj opened 5 years ago
jschultz
commented
5 years ago You need to start the scripts via the shell scripts NormaliseNVPX.sh etc. These do various things including setting up the SQL Anywhere database server, which is the reason why it's falling over when it tries to connect to the server.
atanasj
commented
5 years ago I cannot seem to find these scripts... where are they? When clicking on the hyperlinks I get this error:
jschultz
commented
5 years ago Oh sorry, I incorporated that code into the Python scripts. But I'm not clear which hyperlinks you are referring to in your last message.
Do you get any output when you run the script, or is that error the first thing you see?
Have you installed SQL Anywhere yourself or are you relying on the version packaged with NVivo?
atanasj
commented
5 years ago I only get the error. I just installed SQL Anywhere and ran the the sqlanysrv.sh script:
| => /Users/atanas/OneDrive\ -\ University\ of\ Wollongong/PhD/PR_qual/unzip/nvivotools/helpers/sqlanysrv.sh
/Users/atanas/OneDrive - University of Wollongong/PhD/PR_qual/unzip/nvivotools/helpers/sqlanysrv.sh: line 44: /Applications/SQLAnywhere17/System/bin64/dbeng??: No such file or directoryTrying to convert and I still get the error as above, copied below just in case it is different:
| => python2 NormaliseNVPX.py /Users/atanas/OneDrive\ -\ University\ of\ Wollongong/PhD/PR_qual/unzip/nvivotools-test.nvpx /Users/atanas/OneDrive\ -\ University\ of\ Wollongong/PhD/PR_qual/unzip/nvivotools-test.norm
/Users/atanas/OneDrive - University of Wollongong/PhD/PR_qual/unzip/nvivotools/helpers/sqlanysrv.sh: line 44: /Applications/SQLAnywhere17/System/bin64/dbeng??: No such file or directory
Started database server on port 63722
Traceback (most recent call last):
File "NormaliseNVPX.py", line 149, in <module>
NVivo.Normalise(args)
File "/Users/atanas/OneDrive - University of Wollongong/PhD/PR_qual/unzip/nvivotools/NVivo.py", line 206, in Normalise
nvivoAnnotation = Table('Annotation', nvivomd, autoload=True)
File "<string>", line 2, in __new__
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/deprecations.py", line 130, in warned
return fn(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 496, in __new__
metadata._remove_table(name, schema)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 491, in __new__
table._init(name, metadata, *args, **kw)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 585, in _init
resolve_fks=resolve_fks,
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 626, in _autoload
_extend_on=_extend_on,
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2146, in run_callable
with self._contextual_connect() as conn:
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2226, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect
e, dialect, self
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1536, in _handle_dbapi_exception_noconnection
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 363, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 760, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
rec = pool._do_get()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
self._dec_overflow()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
return self._create_connection()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
self.__connect(first_connect_check=True)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 639, in __connect
connection = pool._invoke_creator(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 451, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 522, in connect
return Connection(args, kwargs)
File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 538, in __init__
parent = Connection.cls_parent = Root("PYTHON")
File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 464, in __init__
'libdbcapi_r.dylib')
File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 456, in load_library
raise InterfaceError("Could not load dbcapi. Tried: " + ','.join(map(str, names)))
sqlalchemy.exc.InterfaceError: (sqlanydb.InterfaceError) (u'Could not load dbcapi. Tried: None,dbcapi.dll,libdbcapi_r.so,libdbcapi_r.dylib', 0)
(Background on this error at: http://sqlalche.me/e/rvf5)OK you shouldn't run sqlanysrv.sh yourself - you need to let the python scripts run it because they supply additional arguments. But still... it should give a clearer error message than that.
atanasj
commented
5 years ago Any suggestions what I can do?
jschultz
commented
5 years ago What happens when you run the script without calling sqlanysrv.sh?
I can try to duplicate the issue when I have time.
On Wednesday, 29 May 2019, atanasj wrote:
Any suggestions what I can do?
-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/BarraQDA/nvivotools/issues/7#issuecomment-49680600
atanasj
commented
5 years ago You mean just running NormaliseNVPX.py? Error info form the SQLAlchemy page here outlined what it is, but I'm not too sure how to address it... Is it not an issues with nvivotools then? Sorry, I'm very much a novice in things git and opensource.
Here's the output, but not sure if I've run it as you asked:
| => /Users/atanas/OneDrive\ -\ University\ of\ Wollongong/PhD/PR_qual/unzip/nvivotools/NormaliseNVPX.py nvivotools-test.nvpx
/Users/atanas/OneDrive - University of Wollongong/PhD/PR_qual/unzip/nvivotools/helpers/sqlanysrv.sh: line 44: /Applications/SQLAnywhere17/System/bin64/dbeng??: No such file or directory
Started database server on port 49302
Traceback (most recent call last):
File "/Users/atanas/OneDrive - University of Wollongong/PhD/PR_qual/unzip/nvivotools/NormaliseNVPX.py", line 149, in <module>
NVivo.Normalise(args)
File "/Users/atanas/OneDrive - University of Wollongong/PhD/PR_qual/unzip/nvivotools/NVivo.py", line 206, in Normalise
nvivoAnnotation = Table('Annotation', nvivomd, autoload=True)
File "<string>", line 2, in __new__
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/deprecations.py", line 130, in warned
return fn(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 496, in __new__
metadata._remove_table(name, schema)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 491, in __new__
table._init(name, metadata, *args, **kw)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 585, in _init
resolve_fks=resolve_fks,
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 626, in _autoload
_extend_on=_extend_on,
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2146, in run_callable
with self._contextual_connect() as conn:
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2226, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect
e, dialect, self
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1536, in _handle_dbapi_exception_noconnection
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 363, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 760, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
rec = pool._do_get()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
self._dec_overflow()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
return self._create_connection()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
self.__connect(first_connect_check=True)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 639, in __connect
connection = pool._invoke_creator(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 451, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 522, in connect
return Connection(args, kwargs)
File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 538, in __init__
parent = Connection.cls_parent = Root("PYTHON")
File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 464, in __init__
'libdbcapi_r.dylib')
File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 456, in load_library
raise InterfaceError("Could not load dbcapi. Tried: " + ','.join(map(str, names)))
sqlalchemy.exc.InterfaceError: (sqlanydb.InterfaceError) (u'Could not load dbcapi. Tried: None,dbcapi.dll,libdbcapi_r.so,libdbcapi_r.dylib', 0)
(Background on this error at: http://sqlalche.me/e/rvf5)OK thanks that's useful. I should be able to find some time to look into this tomorrow and get back to you.
atanasj
commented
5 years ago Great, thanks so much for your help.
jschultz
commented
5 years ago OK I managed to replicate and solve your problem.
In short, and for the record, the problem is related to OSX's System Integrity Protection, which forbids user applications from using shared libraries that are not in the standard location by modifying DYLIB_LIBRARY_PATH For the moment, it is still happy to read dynamic libraries from the current working directory, so we make this work by temporarily changing the working directory to the one containing the SQL Anywhere libraries. Really this is an SQL Anywhere problem and they should come up with a solution.
I also stumbled on another problem related to a bug in the sqlalchemy-sqlany library. You need to install my fork of that library, which fixes the bug. See this pull request.
To pick up these changes you'll need to pull the latest commits, then reinstall sqlalchemy-sqlany either using requirements.txt or by following the (changed) instructions.
atanasj
commented
5 years ago Great, thanks for this. Now, being a novice I will ask some questions that may seem to have obvious answers to the initiated:
requirements.txt to do this? Do I install your fork via pip install https://github.com/BarraQDA/sqlalchemy-sqlany.git or via git? Thanks in advance for your help.
Cheers, Atanas.
jschultz
commented
5 years ago
- How do I temporarily changing the working directory to the one containing the SQL Anywhere libraries?
The updated scripts do all this. All you have to do is refresh your copy of them ('git pull' from the nvivotools directory.
- How would I use the
requirements.txtto do this? Do I install your fork viapip install https://github.com/BarraQDA/sqlalchemy-sqlany.gitor viagit?
The former of these should do the trick. Or you could do 'pip -r requirements.txt' although this might fail on pymssql.
Let me know how you get on.
atanasj
commented
5 years ago Thanks for that. I'm in the middle of some uni work that I need to get out, whilst simultaneously refreshing my Mac after some type of corruption requiring a clean install of macos and reinstalling from a backup image... So, it may take me a several days to get to this but I will definitely let you know how I go.
Cheers, Atanas.
atanasj
commented
5 years ago Okay, so I tried step 2 above, and I got this error:
| => pip2 install https://github.com/BarraQDA/sqlalchemy-sqlany.git
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting https://github.com/BarraQDA/sqlalchemy-sqlany.git
Downloading https://github.com/BarraQDA/sqlalchemy-sqlany.git
| 112kB 1.8MB/s
ERROR: Cannot unpack file /private/var/folders/77/7ht3td2j1vb00csyp1l4wlv40000gn/T/pip-unpack-HFLkZr/sqlalchemy-sqlany.git (downloaded from /private/var/folders/77/7ht3td2j1vb00csyp1l4wlv40000gn/T/pip-req-build-gSpYgK, content-type: text/html; charset=utf-8); cannot detect archive format
ERROR: Cannot determine archive format of /private/var/folders/77/7ht3td2j1vb00csyp1l4wlv40000gn/T/pip-req-build-gSpYgKAny thoughts?
jschultz
commented
5 years ago The command should be
pip install git+git://github.com/BarraQDA/sqlalchemy-sqlanyHi,
I seem to be getting the same error:
/Users/atanas/Library/Scripts/nvivotools/helpers/sqlanysrv.sh: line 44: /Applications/SQLAnywhere17/System/bin64/dbeng??: No such file or directory
Started database server on port 49508
Traceback (most recent call last):
File "NormaliseNVPX.py", line 149, in <module>
NVivo.Normalise(args)
File "/Users/atanas/Library/Scripts/nvivotools/NVivo.py", line 206, in Normalise
nvivoAnnotation = Table('Annotation', nvivomd, autoload=True)
File "<string>", line 2, in __new__
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/deprecations.py", line 130, in warned
return fn(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 496, in __new__
metadata._remove_table(name, schema)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 491, in __new__
table._init(name, metadata, *args, **kw)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 585, in _init
resolve_fks=resolve_fks,
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 626, in _autoload
_extend_on=_extend_on,
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2146, in run_callable
with self._contextual_connect() as conn:
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2226, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect
e, dialect, self
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1486, in _handle_dbapi_exception_noconnection
is_disconnect = dialect.is_disconnect(e, None, None)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy_sqlany/base.py", line 861, in is_disconnect
return e.args[1] in (-101, -308,)
IndexError: tuple index out of rangeDoes it have anything to do with /Applications/SQLAnywhere17/System/bin64/dbeng??: No such file or directory?
jschultz
commented
5 years ago That is quite strange. I'd say the problem is related to the No such file or directory message. But I can't replicate this on my own computer. Can you show me the contents of the directory /Applications/SQLAnywhere17/System/bin64
atanasj
commented
5 years ago Hi, this is what I have in the /Applications/SQLAnywhere17/System/bin64 folder:
.
├── createcert
├── createkey
├── dbbackup
├── dbdsn
├── dbexternc17
├── dbfhide
├── dbisqlc
├── dblocate
├── dbns17
├── dbping
├── dbsupport
├── dbtsinfo
├── dbunload
├── dbupgrad
├── dbvalid
├── dbversion
├── perlenv.pl
├── sa_config.csh
├── sa_config.sh
└── viewcertIt does not contrain the dbeng file, whereas the /Applications/NVivo 12.app/Contents/SQLAnywhere/bin64 folder contains:
.
├── dbeng16
└── dbeng16.licThat's interesting. Where did you install SQLAnywhere from? Can you try again using this link?
You might also want to refresh your copy of the nvivotools repository. I've improved the error handling a bit.
atanasj
commented
5 years ago I'll give it a go and feedback. Do you recommend version 16 or 17?
atanasj
commented
5 years ago Sorry this has taken me so long. So, I have refreshed the repo and just used the NVivo 12.app supplied SQLAnywhere, and still failing. I followed the suggestion by @jamesmnw (see https://github.com/BarraQDA/nvivotools/issues/8#issuecomment-504456728) and symlinked an NVivo.app and then get the below error:
| => python2 /Users/atanas//Library/Scripts/nvivotools/NormaliseNVPX.py ./PR_qual.error-test.nvpx
Started database server on port 49581
Normalising users
Normalising project
Normalising node categories
Normalising nodes
Normalising node attributes
Normalising source categories
Normalising sources
Traceback (most recent call last):
File "/Users/atanas//Library/Scripts/nvivotools/NormaliseNVPX.py", line 154, in <module>
NVivo.Normalise(args)
File "/Users/atanas/Library/Scripts/nvivotools/NVivo.py", line 660, in Normalise
nvivoSource.c.ThumbnailLocation if args.nvivoversion == '12' else nvivoSource.c.Thumbnail,
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/_collections.py", line 212, in __getattr__
raise AttributeError(key)
AttributeError: ThumbnailAny suggestions? Thanks in advance.
jschultz
commented
5 years ago The NormaliseNVPX.py script requires you to tell it which version of NVivo. It should really be able to detect the version, but I haven't written the code to do that. Try adding -nv 12 to your command line.
atanasj
commented
5 years ago Okay. This works now, thanks! However, it failed with the denormalize script. I think this was already raised in issue 8.
jschultz
commented
5 years ago OK thanks for bringing all this to my attention.
I've patched up (De/N)ormaniseNVPX.py to do a better job of looking for SQL Anywhere executables. You can also specify a particular path using the --sqlanywhere argument.
I also added code to auto-detect the NVivo version when normalising a project file.
Let me know how it goes.
I'd also be very interested to know what use you are making of this code. I'm very committed to open source software for qualitative data analysis, and keen to create some kind of community with others. Thinking about making a blog, but in the meantime github is a key channel.
Hi,
I seem to be getting an error when trying to to convert form
.nvpxto.norm. See below:Is there something I'm doing wrong? My system info is:
My
python2version isPython 2.7.16List of
python2librariesI'm not sure what of the above is relevant, but just though I'd include everything. Let me know if you need more info. Cheers.