Open atanasj opened 5 years ago
You need to start the scripts via the shell scripts NormaliseNVPX.sh etc. These do various things including setting up the SQL Anywhere database server, which is the reason why it's falling over when it tries to connect to the server.
I cannot seem to find these scripts... where are they? When clicking on the hyperlinks I get this error:
Oh sorry, I incorporated that code into the Python scripts. But I'm not clear which hyperlinks you are referring to in your last message.
Do you get any output when you run the script, or is that error the first thing you see?
Have you installed SQL Anywhere yourself or are you relying on the version packaged with NVivo?
I only get the error. I just installed SQL Anywhere
and ran the the sqlanysrv.sh
script:
| => /Users/atanas/OneDrive\ -\ University\ of\ Wollongong/PhD/PR_qual/unzip/nvivotools/helpers/sqlanysrv.sh
/Users/atanas/OneDrive - University of Wollongong/PhD/PR_qual/unzip/nvivotools/helpers/sqlanysrv.sh: line 44: /Applications/SQLAnywhere17/System/bin64/dbeng??: No such file or directory
Trying to convert and I still get the error as above, copied below just in case it is different:
| => python2 NormaliseNVPX.py /Users/atanas/OneDrive\ -\ University\ of\ Wollongong/PhD/PR_qual/unzip/nvivotools-test.nvpx /Users/atanas/OneDrive\ -\ University\ of\ Wollongong/PhD/PR_qual/unzip/nvivotools-test.norm
/Users/atanas/OneDrive - University of Wollongong/PhD/PR_qual/unzip/nvivotools/helpers/sqlanysrv.sh: line 44: /Applications/SQLAnywhere17/System/bin64/dbeng??: No such file or directory
Started database server on port 63722
Traceback (most recent call last):
File "NormaliseNVPX.py", line 149, in <module>
NVivo.Normalise(args)
File "/Users/atanas/OneDrive - University of Wollongong/PhD/PR_qual/unzip/nvivotools/NVivo.py", line 206, in Normalise
nvivoAnnotation = Table('Annotation', nvivomd, autoload=True)
File "<string>", line 2, in __new__
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/deprecations.py", line 130, in warned
return fn(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 496, in __new__
metadata._remove_table(name, schema)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 491, in __new__
table._init(name, metadata, *args, **kw)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 585, in _init
resolve_fks=resolve_fks,
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 626, in _autoload
_extend_on=_extend_on,
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2146, in run_callable
with self._contextual_connect() as conn:
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2226, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect
e, dialect, self
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1536, in _handle_dbapi_exception_noconnection
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 363, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 760, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
rec = pool._do_get()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
self._dec_overflow()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
return self._create_connection()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
self.__connect(first_connect_check=True)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 639, in __connect
connection = pool._invoke_creator(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 451, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 522, in connect
return Connection(args, kwargs)
File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 538, in __init__
parent = Connection.cls_parent = Root("PYTHON")
File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 464, in __init__
'libdbcapi_r.dylib')
File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 456, in load_library
raise InterfaceError("Could not load dbcapi. Tried: " + ','.join(map(str, names)))
sqlalchemy.exc.InterfaceError: (sqlanydb.InterfaceError) (u'Could not load dbcapi. Tried: None,dbcapi.dll,libdbcapi_r.so,libdbcapi_r.dylib', 0)
(Background on this error at: http://sqlalche.me/e/rvf5)
OK you shouldn't run sqlanysrv.sh yourself - you need to let the python scripts run it because they supply additional arguments. But still... it should give a clearer error message than that.
Any suggestions what I can do?
What happens when you run the script without calling sqlanysrv.sh?
I can try to duplicate the issue when I have time.
On Wednesday, 29 May 2019, atanasj wrote:
Any suggestions what I can do?
-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/BarraQDA/nvivotools/issues/7#issuecomment-49680600
You mean just running NormaliseNVPX.py
? Error info form the SQLAlchemy
page here outlined what it is, but I'm not too sure how to address it... Is it not an issues with nvivotools
then? Sorry, I'm very much a novice in things git
and opensource.
Here's the output, but not sure if I've run it as you asked:
| => /Users/atanas/OneDrive\ -\ University\ of\ Wollongong/PhD/PR_qual/unzip/nvivotools/NormaliseNVPX.py nvivotools-test.nvpx
/Users/atanas/OneDrive - University of Wollongong/PhD/PR_qual/unzip/nvivotools/helpers/sqlanysrv.sh: line 44: /Applications/SQLAnywhere17/System/bin64/dbeng??: No such file or directory
Started database server on port 49302
Traceback (most recent call last):
File "/Users/atanas/OneDrive - University of Wollongong/PhD/PR_qual/unzip/nvivotools/NormaliseNVPX.py", line 149, in <module>
NVivo.Normalise(args)
File "/Users/atanas/OneDrive - University of Wollongong/PhD/PR_qual/unzip/nvivotools/NVivo.py", line 206, in Normalise
nvivoAnnotation = Table('Annotation', nvivomd, autoload=True)
File "<string>", line 2, in __new__
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/deprecations.py", line 130, in warned
return fn(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 496, in __new__
metadata._remove_table(name, schema)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 491, in __new__
table._init(name, metadata, *args, **kw)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 585, in _init
resolve_fks=resolve_fks,
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 626, in _autoload
_extend_on=_extend_on,
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2146, in run_callable
with self._contextual_connect() as conn:
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2226, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect
e, dialect, self
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1536, in _handle_dbapi_exception_noconnection
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 363, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 760, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
rec = pool._do_get()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
self._dec_overflow()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
return self._create_connection()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
self.__connect(first_connect_check=True)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 639, in __connect
connection = pool._invoke_creator(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 451, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 522, in connect
return Connection(args, kwargs)
File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 538, in __init__
parent = Connection.cls_parent = Root("PYTHON")
File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 464, in __init__
'libdbcapi_r.dylib')
File "/usr/local/lib/python2.7/site-packages/sqlanydb.py", line 456, in load_library
raise InterfaceError("Could not load dbcapi. Tried: " + ','.join(map(str, names)))
sqlalchemy.exc.InterfaceError: (sqlanydb.InterfaceError) (u'Could not load dbcapi. Tried: None,dbcapi.dll,libdbcapi_r.so,libdbcapi_r.dylib', 0)
(Background on this error at: http://sqlalche.me/e/rvf5)
OK thanks that's useful. I should be able to find some time to look into this tomorrow and get back to you.
Great, thanks so much for your help.
OK I managed to replicate and solve your problem.
In short, and for the record, the problem is related to OSX's System Integrity Protection, which forbids user applications from using shared libraries that are not in the standard location by modifying DYLIB_LIBRARY_PATH
For the moment, it is still happy to read dynamic libraries from the current working directory, so we make this work by temporarily changing the working directory to the one containing the SQL Anywhere libraries. Really this is an SQL Anywhere problem and they should come up with a solution.
I also stumbled on another problem related to a bug in the sqlalchemy-sqlany
library. You need to install my fork of that library, which fixes the bug. See this pull request.
To pick up these changes you'll need to pull the latest commits, then reinstall sqlalchemy-sqlany
either using requirements.txt
or by following the (changed) instructions.
Great, thanks for this. Now, being a novice I will ask some questions that may seem to have obvious answers to the initiated:
requirements.txt
to do this? Do I install your fork via pip install https://github.com/BarraQDA/sqlalchemy-sqlany.git
or via git
? Thanks in advance for your help.
Cheers, Atanas.
- How do I temporarily changing the working directory to the one containing the SQL Anywhere libraries?
The updated scripts do all this. All you have to do is refresh your copy of them ('git pull' from the nvivotools directory.
- How would I use the
requirements.txt
to do this? Do I install your fork viapip install https://github.com/BarraQDA/sqlalchemy-sqlany.git
or viagit
?
The former of these should do the trick. Or you could do 'pip -r requirements.txt' although this might fail on pymssql.
Let me know how you get on.
Thanks for that. I'm in the middle of some uni work that I need to get out, whilst simultaneously refreshing my Mac after some type of corruption requiring a clean install of macos
and reinstalling from a backup image... So, it may take me a several days to get to this but I will definitely let you know how I go.
Cheers, Atanas.
Okay, so I tried step 2 above, and I got this error:
| => pip2 install https://github.com/BarraQDA/sqlalchemy-sqlany.git
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting https://github.com/BarraQDA/sqlalchemy-sqlany.git
Downloading https://github.com/BarraQDA/sqlalchemy-sqlany.git
| 112kB 1.8MB/s
ERROR: Cannot unpack file /private/var/folders/77/7ht3td2j1vb00csyp1l4wlv40000gn/T/pip-unpack-HFLkZr/sqlalchemy-sqlany.git (downloaded from /private/var/folders/77/7ht3td2j1vb00csyp1l4wlv40000gn/T/pip-req-build-gSpYgK, content-type: text/html; charset=utf-8); cannot detect archive format
ERROR: Cannot determine archive format of /private/var/folders/77/7ht3td2j1vb00csyp1l4wlv40000gn/T/pip-req-build-gSpYgK
Any thoughts?
The command should be
pip install git+git://github.com/BarraQDA/sqlalchemy-sqlany
Hi,
I seem to be getting the same error:
/Users/atanas/Library/Scripts/nvivotools/helpers/sqlanysrv.sh: line 44: /Applications/SQLAnywhere17/System/bin64/dbeng??: No such file or directory
Started database server on port 49508
Traceback (most recent call last):
File "NormaliseNVPX.py", line 149, in <module>
NVivo.Normalise(args)
File "/Users/atanas/Library/Scripts/nvivotools/NVivo.py", line 206, in Normalise
nvivoAnnotation = Table('Annotation', nvivomd, autoload=True)
File "<string>", line 2, in __new__
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/deprecations.py", line 130, in warned
return fn(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 496, in __new__
metadata._remove_table(name, schema)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 491, in __new__
table._init(name, metadata, *args, **kw)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 585, in _init
resolve_fks=resolve_fks,
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 626, in _autoload
_extend_on=_extend_on,
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2146, in run_callable
with self._contextual_connect() as conn:
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2226, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect
e, dialect, self
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1486, in _handle_dbapi_exception_noconnection
is_disconnect = dialect.is_disconnect(e, None, None)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy_sqlany/base.py", line 861, in is_disconnect
return e.args[1] in (-101, -308,)
IndexError: tuple index out of range
Does it have anything to do with /Applications/SQLAnywhere17/System/bin64/dbeng??: No such file or directory
?
That is quite strange. I'd say the problem is related to the No such file or directory
message. But I can't replicate this on my own computer. Can you show me the contents of the directory /Applications/SQLAnywhere17/System/bin64
Hi, this is what I have in the /Applications/SQLAnywhere17/System/bin64
folder:
.
├── createcert
├── createkey
├── dbbackup
├── dbdsn
├── dbexternc17
├── dbfhide
├── dbisqlc
├── dblocate
├── dbns17
├── dbping
├── dbsupport
├── dbtsinfo
├── dbunload
├── dbupgrad
├── dbvalid
├── dbversion
├── perlenv.pl
├── sa_config.csh
├── sa_config.sh
└── viewcert
It does not contrain the dbeng
file, whereas the /Applications/NVivo 12.app/Contents/SQLAnywhere/bin64
folder contains:
.
├── dbeng16
└── dbeng16.lic
That's interesting. Where did you install SQLAnywhere from? Can you try again using this link?
You might also want to refresh your copy of the nvivotools repository. I've improved the error handling a bit.
I'll give it a go and feedback. Do you recommend version 16 or 17?
Sorry this has taken me so long. So, I have refreshed the repo and just used the NVivo 12.app
supplied SQLAnywhere
, and still failing. I followed the suggestion by @jamesmnw (see https://github.com/BarraQDA/nvivotools/issues/8#issuecomment-504456728) and symlinked an NVivo.app
and then get the below error:
| => python2 /Users/atanas//Library/Scripts/nvivotools/NormaliseNVPX.py ./PR_qual.error-test.nvpx
Started database server on port 49581
Normalising users
Normalising project
Normalising node categories
Normalising nodes
Normalising node attributes
Normalising source categories
Normalising sources
Traceback (most recent call last):
File "/Users/atanas//Library/Scripts/nvivotools/NormaliseNVPX.py", line 154, in <module>
NVivo.Normalise(args)
File "/Users/atanas/Library/Scripts/nvivotools/NVivo.py", line 660, in Normalise
nvivoSource.c.ThumbnailLocation if args.nvivoversion == '12' else nvivoSource.c.Thumbnail,
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/_collections.py", line 212, in __getattr__
raise AttributeError(key)
AttributeError: Thumbnail
Any suggestions? Thanks in advance.
The NormaliseNVPX.py
script requires you to tell it which version of NVivo. It should really be able to detect the version, but I haven't written the code to do that. Try adding -nv 12
to your command line.
Okay. This works now, thanks! However, it failed with the denormalize
script. I think this was already raised in issue 8.
OK thanks for bringing all this to my attention.
I've patched up (De/N)ormaniseNVPX.py to do a better job of looking for SQL Anywhere executables. You can also specify a particular path using the --sqlanywhere
argument.
I also added code to auto-detect the NVivo version when normalising a project file.
Let me know how it goes.
I'd also be very interested to know what use you are making of this code. I'm very committed to open source software for qualitative data analysis, and keen to create some kind of community with others. Thinking about making a blog, but in the meantime github is a key channel.
Hi,
I seem to be getting an error when trying to to convert form
.nvpx
to.norm
. See below:Is there something I'm doing wrong? My system info is:
My
python2
version isPython 2.7.16
List of
python2
librariesI'm not sure what of the above is relevant, but just though I'd include everything. Let me know if you need more info. Cheers.