MLCP Test Failure

[twcook]

The _test_mlcpdownload.py doesn't run because the files that are imported from marklogic/tools and marklogic/recipes are missing.

[paul-hoehne]

You might need to pull/rebase. The relevant classes should be there (defined in the init.py files).

[twcook]

Ah, I didn't see the files and didn't look in the init.py

I assumed that is why I am getting this traceback:

´´´

(S3Model)tim@tim-dev:~/S3Model/MarkLogic_Python$ python -m unittest tests/tools/test_mlcp_download.py Traceback (most recent call last): File "/usr/local/lib/python3.4/runpy.py", line 170, in _run_module_as_main "main", mod_spec) File "/usr/local/lib/python3.4/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.4/unittest/main.py", line 18, in main(module=None) File "/usr/local/lib/python3.4/unittest/main.py", line 92, in init self.parseArgs(argv) File "/usr/local/lib/python3.4/unittest/main.py", line 139, in parseArgs self.createTests() File "/usr/local/lib/python3.4/unittest/main.py", line 146, in createTests self.module) File "/usr/local/lib/python3.4/unittest/loader.py", line 146, in loadTestsFromNames suites = [self.loadTestsFromName(name, module) for name in names] File "/usr/local/lib/python3.4/unittest/loader.py", line 146, in suites = [self.loadTestsFromName(name, module) for name in names] File "/usr/local/lib/python3.4/unittest/loader.py", line 114, in loadTestsFromName parent, obj = obj, getattr(obj, part) AttributeError: 'module' object has no attribute 'test_mlcp_download'

´´´

Missing imports can cause that.

I'll troubleshoot further.

[paul-hoehne]

I opened a couple of bugs. I'll start tackling them once I get a Windows environment set up. I have a couple of things to do this afternoon (UTC -5:00) but I'll try to get at least one of the bugs squashed today.

[paul-hoehne]

I think I fixed the downloader test. Take a look and see if it works for you.

https://github.com/paul-hoehne/MarkLogic_Python/pull/17

[twcook]

It looks like your fix works.

I thought that Content Pump was installed by default. So, I need to go install it.

[paul-hoehne]

My point of having the MLCP loader class is that it would download and install the current version of content pump in a .mlcp directory. Let's say you're new to MarkLogic and want to create a database and load some data. I don't want you to have to stop, download MLCP, install it, add it to your path, make sure it works, and then get back to loading data. I don't want the user to have to worry about content pump.

[twcook]

That sounds great. Okay, then so I should not get the missing class error? (see below) I see that it downloaded mlcp and setup the directories.

My guess is that it hs something to do with setting up the CLASSPATH? I am not a Java guru but I'll take a look at it later.

However, does the test have to run on the same machine as the server? I see that, "Distributed mode is only supported on 64-bit Linux." Which in my case would be great since that is my OS.

TRACEBACK

...mlcp/mlcp/bin/mlcp.sh: 3: .mlcp/mlcp/bin/mlcp.sh: Bad substitution Error: Could not find or load main class com.marklogic.contentpump.ContentPump Pausing 15 seconds for server restart

F

FAIL: test_load_data (tests.tools.test_mlcp_download.TestMLCPDownload)

Traceback (most recent call last): File "/home/tim/S3Model/MarkLogic_Python/tests/tools/test_mlcp_download.py", line 67, in test_load_data self.assertIsNotNone(exampledb[u'content'].get_document(conn, "/test/data1/purchases/december/purchase-001.json")) AssertionError: unexpectedly None

---

Ran 3 tests in 100.173s

FAILED (failures=1)

On Mon, Apr 6, 2015 at 11:02 AM, Paul Hoehne notifications@github.com wrote:

My point of having the MLCP loader class is that it would download and install the current version of content pump in a .mlcp directory. Let's say you're new to MarkLogic and want to create a database and load some data. I don't want you to have to stop, download MLCP, install it, add it to your path, make sure it works, and then get back to loading data. I don't want the user to have to worry about content pump.

— Reply to this email directly or view it on GitHub https://github.com/paul-hoehne/MarkLogic_Python/issues/11#issuecomment-90074253 .

Timothy Cook LinkedIn Profile:http://www.linkedin.com/in/timothywaynecook MLHIM http://www.mlhim.org

[paul-hoehne]

That shouldn't happen. What should be happening is:

1. In the current working directory a .mlcp directory is created
2. Download the current version of MLCP.
3. Unpack that into .mlcp/mlcp
4. Execute .mlcp/mlcp/bin/mlcp.sh ("sh .mlcp/mlcp/bin/mlcp.sh")

That should be enough. I think mlcp.sh figures out where the jars live based on the absolute location of the of the mlcp.sh file. In fact, I'm using the "sh" shell to run it. Unless the shell is the issue. Try replacing sh on https://github.com/paul-hoehne/MarkLogic_Python/blob/master/marklogic/tools/__init__.py#L77 with "bash".

[twcook]

FYI:

That error msg. may be a bit misleading. I commented out the teardown code in the test_load_data function. The AppServer, database and forests are being created. When I explore the database in qconsole there are no documents there.

HTH.

[paul-hoehne]

This is the MLCP script. It's not running because like 3 DIR="$( cd "$( dirname "${BASH_SOURCE[0]}")" && pwd)"/../ is failing. That means it's not able to set the DIR variable, which causes the LIB_HOME variable to not be set, which means the files that should be in $DIR/lib don't get added to the CLASSPATH. I'm getting this from the trace you posted but can't repro the issue on my end. Are you running on Linux, Mac OS or on Windows with Cygwin?

#!/bin/bash
unset CLASSPATH
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"/../
VMARGS="-DCONTENTPUMP_HOME=$DIR/lib -Dfile.encoding=UTF-8" 
LIB_HOME=$DIR/lib

for file in ${LIB_HOME}/*.jar
do
  if [ ! -z "$CLASSPATH" ]; then
    CLASSPATH=${CLASSPATH}":"$file
  else
    CLASSPATH=$file
  fi
done
CLASSPATH=$DIR/conf:$CLASSPATH
java -cp $CLASSPATH $VMARGS $JVM_OPTS com.marklogic.contentpump.ContentPump $*

[twcook]

The client is Ubuntu 14.04 the server is Windows 8.1

Two different machines.

On Mon, Apr 6, 2015 at 12:13 PM, Paul Hoehne notifications@github.com wrote:

This is the MLCP script. It's not running because like 3 DIR="$( cd "$( dirname "${BASH_SOURCE[0]}")" && pwd)"/../ is failing. That means it's not able to set the DIR variable, which causes the LIB_HOME variable to not be set, which means the files that should be in $DIR/lib don't get added to the CLASSPATH. I'm getting this from the trace you posted but can't repro the issue on my end. Are you running on Linux, Mac OS or on Windows with Cygwin?

!/bin/bash

unset CLASSPATH DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"/../ VMARGS="-DCONTENTPUMP_HOME=$DIR/lib -Dfile.encoding=UTF-8" LIB_HOME=$DIR/lib

for file in ${LIBHOME}/.jar do if [ ! -z "$CLASSPATH" ]; then CLASSPATH=${CLASSPATH}":"$file else CLASSPATH=$file fi done CLASSPATH=$DIR/conf:$CLASSPATH java -cp $CLASSPATH $VMARGS $JVMOPTS com.marklogic.contentpump.ContentPump $

— Reply to this email directly or view it on GitHub https://github.com/paul-hoehne/MarkLogic_Python/issues/11#issuecomment-90097971 .

Timothy Cook LinkedIn Profile:http://www.linkedin.com/in/timothywaynecook MLHIM http://www.mlhim.org

[paul-hoehne]

Thanks. I'm going to set up Ubuntu 14.04 so I can try to repro the issue.

[twcook]

Any progress on this? If you do not have time to work on it now maybe I can take a look this next week.

[paul-hoehne]

I set up a VM with Ubuntu 14.4 but haven't tried running the examples. The one thing I would try is manually running mlcp from that directory and see if the script fails. I would also try running with "sh .mlcp/bin/mlcp.sh" and just ".mlcp/bin/mlcp.sh".

[twcook]

Ubuntu points to dash for the default sh. Executing the script with bash works as expected. See this SO question http://stackoverflow.com/questions/20615217/bash-bad-substitution

On Mon, Apr 13, 2015 at 10:49 AM, Paul Hoehne notifications@github.com wrote:

I set up a VM with Ubuntu 14.4 but haven't tried running the examples. The one thing I would try is manually running mlcp from that directory and see if the script fails. I would also try running with "sh .mlcp/bin/mlcp.sh" and just ".mlcp/bin/mlcp.sh".

— Reply to this email directly or view it on GitHub https://github.com/paul-hoehne/MarkLogic_Python/issues/11#issuecomment-92360556 .

Timothy Cook LinkedIn Profile:http://www.linkedin.com/in/timothywaynecook MLHIM http://www.mlhim.org

[twcook]

I suggest that you set the execute permission on the script.

[paul-hoehne]

Thanks! My plan is to only download mlcp if it's not in the user's executable path. I'm thinking of setting up a trello board to track the issue and features in which the project is interested.

[paul-hoehne]

https://github.com/paul-hoehne/MarkLogic_Python/pull/21

It should only download MLCP if it can't find mlcp/bin in the path. It will also attempt to set the execute bits on the mlcp.sh file.