Wooey issue

[OliJimbo]

Hi there,

I'm very excited to be able to convert my Nvivo projects into something more open and available. I have been trying to use Wooey to convert my NVPX files and at the moment it just says waiting. I initially tried with a very large file and have just tried a minimal example (one source, one node) to see if it was a size issue but the files have still to been processed and the console is completely blank. I wondered whether you were still hosting the files on your server? So far I've tried to do this on a Mac and have used Firefox and Safari

Very best wishes,

Oliver

[jschultz]

Hi Oliver, Thanks for the feedback. I found the problem causing the blank console, which was that Wooey wasn't starting because of a corrupted internal database, I think most probably caused by my server rebooting at some point. I've fixed that problem. Next I found your small file sitting on the server - or at least the one called "Untitled.nvpx" For some reason SQL Anywhere is failing to open that file - it reports 'Cannot open transaction log file'. I'll need to look into that more closely, but is it possible that you uploaded it while it was already open in NVivo? Perhaps you could try it again and see if it works better? Cheers, Jonathan

[jschultz]

Update: I've managed to circumvent the log file issue. Now it seems that the structure of your file is slightly changed from what my script expects. Which version of NVivo did you use to create it? NVivotools currently only supports versions 10 and 11. QSR have the very annoying habit of changing their file formats when they release new versions on NVivo.

[OliJimbo]

Ah - Sorry, I thought I was running 10 but it's actually 11. If there are any novice jobs that could help progress this project to include version 11 I'll happily take a look - but I've taken a quick look at your Normalise function (which I assume is the one that would need updating) and it looks like the version 11 NVPX files need to be unpacked which I doubt I could do! In any case, it looks like Wooey is now working so I'll close the issue. Many thanks for your time!

[jschultz]

A couple of things:

First, NVivotools is supposed to handle NVivo 11, and thus far it has. But it looks like your files were created by a more recent version of NVivo 11 than I have used. Can you check the precise version of NVivo you are using?

It seems that apart from the issue of log files, the file you sent me stores the source objects (ie the Word doc, PDF or whatever) in a separate table (I forget the name, something like BlobData) rather than in a column of the table "Source". The quick and dirty way of getting around this would be just to ignore that field (you still have the textual content of the source in a field called "PlainText"). But it wouldn't take much work to look up the content in the new table, but would just require a little attention to detail to handle the various different database structures used by different NVivo versions.

So if you're interested stick with me and we'll get this working.

[OliJimbo]

Hi Jonathan,

I'm very interested and thank you for following up.

The version of NVIvo I have is 11.4.3(2084). If there is anything else I can do (upload stuff, send you new files etc) to help please let me know.

Also, I had wondered why nodes and cases were stored in plain text when they exist as internals in PDF form - its interesting to know that Nvivo extracts all of the data separately!

Best wishes,

Oliver

[jschultz]

So I finally managed to check which version of NVivo I was using, it was 11.2.2 (1707). I'll need to install an update and see what has changed.

I'd say source is stored as plaintext as well as formatted to improve performance. PDFs in particular are quite difficult to manage because they contain no information about text flow so it has to be guessed.

[jschultz]

Hi again Oliver,

It took a while but I have now updated and tested the Normalise script with projects created by NVivo 11.4.3 for Mac, and installed the update on wooey.barraqda.org

Note that you need to make sure that the NVivo project is not open in NVivo - that seemed to be the cause of the first error you were getting where the script couldn't even open the database.

Let me know how you get on. I use these scripts for myself fairly regularly but I'd love to see it working for others.

Cheers, Jonathan

[OliJimbo]

Brilliant! Thank you so much for that Jonathan. I'll give it a try later this evening and let you know.

[OliJimbo]

Hi again Jonathan,

The Normalise script does indeed work! However, I am now getting the log file complaint from the NVPX2RQDA script - I'm not sure if the following error message will help at all, but I've included it.

In the meantime, I'll see if I can get it working from my console by going from Normalise->RQDA.

I think there is some real potential for data sharing with this workflow - RQDA is a pain to install and I currently can only get it working on a VM running the latest Ubuntu release - hopefully it will get added to CRAN at some point - but I think the GTK is the main issue.

Best wishes,

Oliver

Cannot open transaction log file -- No such file or directory Traceback (most recent call last): File "/home/barraqda/wooey/nvivotools/nvivotools/user_uploads/wooey_scripts/NVPX2RQDA_ex8jpSH_Uw8ODWF.py", line 109, in <module> NVivo.Normalise(args) File "/home/barraqda/wooey/nvivotools/nvivotools/user_uploads/wooey_scripts/NVivo.py", line 205, in Normalise nvivoAnnotation = Table('Annotation', nvivomd, autoload=True) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/schema.py", line 416, in __new__ metadata._remove_table(name, schema) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__ compat.reraise(exc_type, exc_value, exc_tb) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/schema.py", line 411, in __new__ table._init(name, metadata, *args, **kw) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/schema.py", line 484, in _init self._autoload(metadata, autoload_with, include_columns) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/schema.py", line 508, in _autoload self, include_columns, exclude_columns File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1971, in run_callable with self.contextual_connect() as conn: File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect self._wrap_pool_connect(self.pool.connect, None), File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect e, dialect, self) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception_noconnection exc_info File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect return fn() File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect return _ConnectionFairy._checkout(self) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout fairy = _ConnectionRecord.checkout(pool) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout rec = pool._do_get() File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1060, in _do_get self._dec_overflow() File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__ compat.reraise(exc_type, exc_value, exc_tb) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1057, in _do_get return self._create_connection() File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 323, in _create_connection return _ConnectionRecord(self) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 449, in __init__ self.connection = self.__connect() File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect connection = self.__pool._invoke_creator(self) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect return dialect.connect(*cargs, **cparams) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect return self.dbapi.connect(*cargs, **cparams) File "/usr/local/lib/python2.7/dist-packages/sqlanydb.py", line 522, in connect return Connection(args, kwargs) File "/usr/local/lib/python2.7/dist-packages/sqlanydb.py", line 584, in __init__ self.handleerror(*error) File "/usr/local/lib/python2.7/dist-packages/sqlanydb.py", line 596, in handleerror eh(self, None, errorclass, errorvalue, sqlcode) File "/usr/local/lib/python2.7/dist-packages/sqlanydb.py", line 379, in standardErrorHandler raise errorclass(errorvalue,sqlcode) sqlalchemy.exc.OperationalError: (sqlanydb.OperationalError) ('Database server not found', -100)

[jschultz]

Great to hear that Normalise is working again for you!

That error message, especially the 'Cannot open transaction log file' bit indicates to me that the NVPX file you have uploaded is still open somewhere else (probably in NVivo itself). Can you check that this isn't the case?

It is possible to have the script still function (by using dblog -n to remove the transaction log from the database) but that would require some rearrangements of my scripts, so for the moment it is in the 'would be nice' basket.

[OliJimbo]

Hi Jonathan,

Nvivo was not open at the time but had been recently. I've just retried and there is now a different error message suggesting that a background process may have been running.

The normalisation seems to be completing but is returning one of the CreatedBy entries as a nonetype.

Normalising users Normalising project Normalising node categories Normalising nodes Normalising node attributes Normalising source categories Normalising sources Normalising source attributes Normalising taggings Normalising annotations Converting project Traceback (most recent call last): File "/home/barraqda/wooey/nvivotools/nvivotools/user_uploads/wooey_scripts/NVPX2RQDA_ex8jpSH_Uw8ODWF.py", line 121, in <module> RQDA.Norm2RQDA(args) File "/home/barraqda/wooey/nvivotools/nvivotools/user_uploads/wooey_scripts/RQDA.py", line 297, in Norm2RQDA normUser.c.Id == normSourceCategory.c.CreatedBy TypeError: 'NoneType' object is not iterable

[jschultz]

Ah very interesting. I guess the script NVPX2RQDA could be made more robust against this kind of thing. Shouldn't be too hard if you feel up to it. Otherwise I'll take a look when I have a free moment, probably next week at this stage.

[jschultz]

I just had a quick look and found a stupid error. Should work now.

[OliJimbo]

Brilliant - I've just opened a file that I converted on Wooey in RQDA and it all works! Thank you very much for that!