Integration infrastructure

[JohannesGaessler]

Fixes https://github.com/DistributedTaskScheduling/JobAdder/issues/113.

I have done relatively little work on this PR because I was waiting for more progress to be done in https://github.com/DistributedTaskScheduling/JobAdder/pull/108 and https://github.com/DistributedTaskScheduling/JobAdder/pull/112. To make the worker main usable I applied some quick, hacky fixes. On the worker main PR the JobWorker argument in the execute methods should be replaced with DockerInterface.

[nikolatzotchev]

I have a few questions

1. I was getting some errors with the database, I could log in without having a password to the user, I just set up 1234 and it works, perhaps it has something to do with the fact that I have edited the pg_hba.conf, can you please show me yours so that I can copy it.

2. Now the stuff with the database works, but I'm getting ~~paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 127.0.0.1 or ::1 Do you have any clue what is wrong.~~ I fixed that but now I'm getting [Errno 111] Connection refused and then an assert fails.

[ammen99]

I have tried to start this and got the same problems as @nikolatzotchev . I found what the problem is:

We are using test.integration..... However, it seems that the name test can conflict with https://docs.python.org/3/library/test.html and if both are present, then python thinks we are trying to access something in this library.

As a workaround, I suggest that we move the integration tests to a third folder src/ja_integration so that we are sure we don't have any such conflicts.

[JohannesGaessler]

I pushed a new version where I moved the remotes to ja_integration.remote. Please confirm whether this works on your machine @ammen99 @nikolatzotchev .

[nikolatzotchev]

It works on my pc.

[ammen99]

It doesn't on mine :( It still complains about the import test...)

[JohannesGaessler]

@ammen99 Just to make sure: Did you forget to uninstall JobAdder after you tested the pip installation?

[ammen99]

Yes, I just went through all of the places where pip could install job adder and there is nothing installed.

[ammen99]

Also, please add a password to the database. My postgresql installation does not want to accept connections without password.

[JohannesGaessler]

My postgresql installation does not want to accept connections without password.

Does setting the password in line 105 of test/integration/base.py not work?

[ammen99]

Does setting the password in line 105 of test/integration/base.py not work?

Yes, but I have to do it manually. I think it will not hurt to add it to the PR for ease of testing on our computers (Nikola had the same issue). Plus I assume we're going to write more tests using this infrastructure so we'll need to do that in the future, on our PCs.

[JohannesGaessler]

It doesn't on mine :( It still complains about the import test...)

Please try again with the newest version.

[ammen99]

Please try again with the newest version.

Newest version of what?

[JohannesGaessler]

Forgot to push it, sry.

[ammen99]

@JohannesGaessler Yes, the last commit works when I add the password manually.

[JohannesGaessler]

One more question, out of curiosity, when do we need multiple databases per process?

When doing the integration tests :^) . The setUp method creates a new instance of JobCenter which in turn creates a new instance of SQLDatabase.

[ammen99]

When doing the integration tests :^) . The setUp method creates a new instance of JobCenter which in turn creates a new instance of SQLDatabase.

Isn't that like one database here? And then you can get JobCenter._database instead of creating a new one.

Doesn't matter of course.

Can you rebase this PR so we can merge it?

[JohannesGaessler]

I excluded ja_integration in the install script. Is there something else that I would need to consider for your changes @ammen99 ?

[ammen99]

Good catch. I do not think there is anything else special.