Open psychtek opened 1 year ago
jschultz
commented
1 year ago The call to install_name_tool is in sqlanyenv.sh which is called from NormaliseNVPX.py. After that something must happen that makes env python fail. If you are able to add debug files into NormaliseNVPX.py you ought to be able to track down the problem. Likely it's a Mac thing - there's a bit of hackery in there to make it work which hasn't been tested on recent MacOS releases.
psychtek
commented
1 year ago I started doing a deep dive and I think I found where it was going funky. I'll recreate and post what I come up with.
Has this been tested with Python3? I read somewhere that there's been some changes to how Python reads binary. 🐍
jschultz
commented
1 year ago OK good luck. I updated all the tools for Python 3 after Python 2 was discontinued. There were certainly changes around Unicode strings if that's what you mean. It's conceivable that this has created problems, but wouldn't rate it very highly. One thing in particular that isn't really tested is the NVivo 12 stuff. I've never done a comprehensive upgrade for NVivo 12, just made a few ad hoc changes to solve problems that users reported.
psychtek
commented
1 year ago My python 🐍 is pretty average as I work full time with R, so apologies for the approach. I pulled the guts of the NormaliseNVPX script and worked through each section and recorded it here to see if any of it helps. The previous error must have been a PATH setting as this seems to be a non issue now.
A super phenomenal, thank you on this tool set! It has been a lifesaver for archiving our workflow.
Aside from the warnings there are two ForeignKeyConstraint errors. If I comment these out and 'skip' then I can successfully convert a npx to a norm file.
These two lines produce an error:
>>> nvivoAnnotation = Table('Annotation', nvivomd, autoload=True)
ArgumentError: ForeignKeyConstraint with duplicate source column references are not supported.
>>> nvivoNodeReference = Table('NodeReference', nvivomd, autoload=True)
ArgumentError: ForeignKeyConstraint with duplicate source column references are not supported.When I comment out the two lines from Nvivo.py that are creating the previous issues:
#nvivoAnnotation = Table('Annotation', nvivomd, autoload=True)
nvivoCategory = Table('Category', nvivomd, autoload=True)
nvivoExtendedItem = Table('ExtendedItem', nvivomd, autoload=True)
nvivoItem = Table('Item', nvivomd, autoload=True)
#nvivoNodeReference = Table('NodeReference', nvivomd, autoload=True)
nvivoProject = Table('Project', nvivomd, autoload=True)
nvivoRole = Table('Role', nvivomd, autoload=True)
nvivoSource = Table('Source', nvivomd, autoload=True)
nvivoUserProfile = Table('UserProfile', nvivomd, autoload=True)
And then skip these in the arguments, I get a .norm file:
❯ python3 NormaliseNVPX.py -nv 12 -a 'skip' -t 'skip' ../../../Nvivo\ Projects/repliCATS_KnownOutcomeClaims_Round3_Merged.nvpx
Found SQLAnywhere installation at /Applications/NVivo 12.app/Contents/SQLAnywhere/bin64
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Applications/NVivo 12.app/Contents/SQLAnywhere/lib64/libdbcapi_r.rpath.dylib
Started database server on port 65332
/usr/local/lib/python3.10/site-packages/sqlalchemy_sqlany/base.py:464: SAWarning: Dialect sqlany:<module 'sqlanydb' from '/usr/local/lib/python3.10/site-packages/sqlanydb.py'> will not make use of SQL compilation caching as it does not set the 'supports_statement_cache' attribute to ``True``. This can have significant performance implications including some performance degradations in comparison to prior SQLAlchemy versions. Dialect maintainers should seek to set this attribute to True after appropriate development and testing for SQLAlchemy 1.4 caching support. Alternatively, this attribute may be set to False which will disable this warning. (Background on this error at: https://sqlalche.me/e/14/cprf)
return connection.scalar(
Normalising users
/Users/awwillc/Repos/RAT/Scripts/NVivo_Extraction/nvivotools/NVivo.py:175: SAWarning: TypeDecorator UUID() will not produce a cache key because the ``cache_ok`` attribute is not set to True. This can have significant performance implications including some performance degradations in comparison to prior SQLAlchemy versions. Set this attribute to True if this type object's state is safe to use in a cache key, or False to disable this warning. (Background on this error at: https://sqlalche.me/e/14/cprf)
for row in conn.execute(select([table.c[column] for column in columns]))]
Normalising project
Normalising node categories
Normalising nodes
Normalising node attributes
Normalising source categories
Normalising sources
Normalising source attributesWhere would I find the Annotation and NodeReference tables? Is this on the project setup side of things in NVivo?
psychtek
commented
1 year ago Reading through #7 to check a few things and came across this post about the argument error so I ran the code as you recommended and returned the following error in the console:
Collecting git+git://github.com/BarraQDA/sqlalchemy-sqlany
Cloning git://github.com/BarraQDA/sqlalchemy-sqlany to /private/var/folders/1m/r7b19ccj54sfvx6ymrlkgd186l0xc5/T/pip-req-build-5z6r51ne
Running command git clone --filter=blob:none --quiet git://github.com/BarraQDA/sqlalchemy-sqlany /private/var/folders/1m/r7b19ccj54sfvx6ymrlkgd186l0xc5/T/pip-req-build-5z6r51ne
fatal: unable to connect to github.com:
github.com[0: 20.248.137.48]: errno=Operation timed out
error: subprocess-exited-with-error
× git clone --filter=blob:none --quiet git://github.com/BarraQDA/sqlalchemy-sqlany /private/var/folders/1m/r7b19ccj54sfvx6ymrlkgd186l0xc5/T/pip-req-build-5z6r51ne did not run successfully.
│ exit code: 128
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× git clone --filter=blob:none --quiet git://github.com/BarraQDA/sqlalchemy-sqlany /private/var/folders/1m/r7b19ccj54sfvx6ymrlkgd186l0xc5/T/pip-req-build-5z6r51ne did not run successfully.
│ exit code: 128
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
oops ok so I needed to change:
pip3 install git+git://github.com/BarraQDA/sqlalchemy-sqlanygit to https to the following:
pip3 install git+https://github.com/BarraQDA/sqlalchemy-sqlany
Installing collected packages: sqlalchemy-sqlany
Attempting uninstall: sqlalchemy-sqlany
Found existing installation: sqlalchemy-sqlany 1.0.4
Uninstalling sqlalchemy-sqlany-1.0.4:
Successfully uninstalled sqlalchemy-sqlany-1.0.4
Successfully installed sqlalchemy-sqlany-1.0.3I than reran as previous without the -skip args set and getting a bunch of other errors now. I am a little lost 😅
Found SQLAnywhere installation at /Applications/NVivo 12.app/Contents/SQLAnywhere/bin64
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Applications/NVivo 12.app/Contents/SQLAnywhere/lib64/libdbcapi_r.rpath.dylib
Started database server on port 50672
Traceback (most recent call last):
File "/Users/awwillc/Repos/RAT/Scripts/NVivo_Extraction/nvivotools/NormaliseNVPX.py", line 165, in <module>
NVivo.Normalise(args)
File "/Users/awwillc/Repos/RAT/Scripts/NVivo_Extraction/nvivotools/NVivo.py", line 212, in Normalise
nvivoAnnotation = Table('Annotation', nvivomd, autoload=True)
File "<string>", line 2, in __new__
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/util/deprecations.py", line 309, in warned
return fn(*args, **kwargs)
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/sql/schema.py", line 616, in __new__
with util.safe_reraise():
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
raise exception
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/sql/schema.py", line 612, in __new__
table._init(name, metadata, *args, **kw)
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/sql/schema.py", line 687, in _init
self._autoload(
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/sql/schema.py", line 720, in _autoload
insp = inspection.inspect(autoload_with)
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/inspection.py", line 64, in inspect
ret = reg(subject)
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/engine/reflection.py", line 182, in _engine_insp
return Inspector._construct(Inspector._init_engine, bind)
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/engine/reflection.py", line 117, in _construct
init(self, bind)
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/engine/reflection.py", line 128, in _init_engine
engine.connect().close()
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/engine/base.py", line 3315, in connect
return self._connection_cls(self, close_with_result=close_with_result)
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
else engine.raw_connection()
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/engine/base.py", line 3394, in raw_connection
return self._wrap_pool_connect(self.pool.connect, _connection)
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/engine/base.py", line 3361, in _wrap_pool_connect
return fn()
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/pool/base.py", line 320, in connect
return _ConnectionFairy._checkout(self)
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/pool/base.py", line 884, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/pool/base.py", line 486, in checkout
rec = pool._do_get()
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/pool/impl.py", line 145, in _do_get
with util.safe_reraise():
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
raise exception
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
return self._create_connection()
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/pool/base.py", line 266, in _create_connection
return _ConnectionRecord(self)
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/pool/base.py", line 381, in __init__
self.__connect()
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/pool/base.py", line 691, in __connect
)._exec_w_sync_on_first_run(self.dbapi_connection, self)
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/event/attr.py", line 320, in _exec_w_sync_on_first_run
self(*args, **kw)
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/event/attr.py", line 334, in __call__
fn(*args, **kw)
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/util/langhelpers.py", line 1695, in go
return once_fn(*arg, **kw)
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/engine/create.py", line 674, in first_connect
dialect.initialize(c)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy_sqlany/base.py", line 470, in initialize
super(SQLAnyDialect, self).initialize(connection)
File "/Users/awwillc/Library/Python/3.10/lib/python/site-packages/sqlalchemy/engine/default.py", line 414, in initialize
self.default_schema_name = self._get_default_schema_name(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy_sqlany/base.py", line 465, in _get_default_schema_name
text("SELECT current user",
TypeError: text() got an unexpected keyword argument 'typemap'
OK you're doing well.
The last problem was simply that I hadn't merged upstream fixes to sqlalchemy into my fork. If you do the pip3 install ... command now you should get past this latest error.
psychtek
commented
1 year ago Ok that all seems to have fixed that, thank you!
I started fresh and then that error returned to haunt me.
env: python: Not a directoryWhen I comment out line 89 that's spawning a new process it runs successfully.
# On non-Windows OS, need to set up environment for SQL Anywhere server and restart process.
if os.name != 'nt':
# Check if already done
if not os.environ.get('_restart'):
if args.sqlanywhere:
os.environ['sqlanywhere'] = args.sqlanywhere
envlines = subprocess.check_output(helperpath + 'sqlanyenv.sh', text=True).splitlines()
for envline in envlines:
env = re.match(r"(?P<name>\w+)=(?P<quote>['\"]?)(?P<value>.*)(?P=quote)", envline, re.MULTILINE | re.DOTALL).groupdict()
os.environ[env['name']] = env['value']
os.environ['_restart'] = 'TRUE'
#os.execve(sys.argv[0], sys.argv, os.environ)
Like you suggested, likely a mac thing. But this seems to be a work around for now for things on my end.
Nvivo version 12 Getting the error below when trying to convert a
nvpxto anormfile. Have read through all the issues and #13 seemed to have more luck but I am unable to get anything to convert.Python Config
Packages:
Converting
Error
And the sqlanywhere config script output
Hopefully this is an easy fix and I am simply missing a step somewhere!? Thanks