bundle custom lib in docker image

[Wxl19980214]

Describe your changes

Add path check. If Path(lib/') exist, add a PYTHONPATH in docerfile to look for packages under lib.

Issue ticket number and link

Closes #85

Checklist before requesting a review

• [x] I have performed a self-review of my code
• [x] I have added thorough tests (when necessary).
• [x] I have added the right documentation (when needed). Product update? If yes, write one line about this update.

[edublancas]

overall, it looks good!

did you get a chance to test it?

Essentially, we wanna make sure that if I have a lib/something.py, I can do:

python

and then:

import something

[edublancas]

please write an automated test, see this for details: https://github.com/ploomber/soopervisor/issues/88

[Wxl19980214]

@edublancas  Yes. I have tested in this way. This is my project structure:
project/
        -> lib/ 
                  ->package_a.py (contains function printTest() )
        -> fit.py  (from lib.package_a import printTest )
        -> pipeline.yaml
        -> soopervisor.yaml
        ....

And I use sooperviosr add and export to build the image. It looks like there's no error when I run the image. 

[idomic]

@Wxl19980214 please add a few tests.

[Wxl19980214]

@edublancas @idomic I am having troubling writing test: Here is what I have:

@pytest.mark.parametrize('CLASS_', [
    AWSBatchExporter,
    AirflowExporter,
    ArgoWorkflowsExporter,
    KubeflowExporter,
],
                         ids=[
                             'batch',
                             'airflow',
                             'argo',
                             'kubeflow',
                         ])
def test_docker_local_lib_import(
    mock_docker_and_batch,
    tmp_sample_project,
    no_sys_modules_cache,
    skip_repo_validation,
    capsys,
    CLASS_,
):

    exporter = CLASS_.new(path_to_config='soopervisor.yaml', env_name='serve')

    exporter.add()
    exporter.export(mode='force', skip_tests=True)

    client = docker.from_env()
    client.containers.run('python', 'import package_a')

    captured = capsys.readouterr()

    print(captured.out)

I have added the lib/ directory with package_a under asset/sample_project. However, it seems like the docker.from_env() is mocked so I cannot get real result. Any ideas?

[Wxl19980214]

Also for some reason, i cannot recreate what I had last week. After soopervisor export, what command do we use to run the image? docker run example:latest? I have tried and it looks like not working

[idomic]

Maybe don't mock docker.from_env() ? I don't see a way around it if you'd like to check what's inside the container and run commands in it.

You can run the image interactively with the -it flag, is that what you're talking about? You can also use the commander and run something like this:

e.run('docker',
              'run',
              image_local,
              'ploomber',
              'status',
              '--entry-point',
              entry_point,
              description='Testing image',
              error_message='Error while testing your docker image with')

[Wxl19980214]

I mean after soopervisor export argo --skip-tests --ignore-git --mode force I should get a docker image. Last week I can run some command to actually run this image, and in there I can directly test by python, import package_a. But today it looks like I cannot activate image.

[idomic]

That's right. After the export command you should have the image tagged locally and you can simply use docker run tag -it to run it and debug. You can also list the available images to see if it's not there for some reason.

[Wxl19980214]

Ok this is weird, I have checked I have images locally by docker images

example $ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE example latest af4bedf7589c 2 hours ago 1.15GB

but when I do: docker run example -it, it returns this: [FATAL tini (6)] exec -it failed: No such file or directory

[idomic]

It should be something like this docker run image_name:tag_name

On Mon, Jun 27, 2022 at 2:35 PM Xilin @.***> wrote:

Ok this is weird, I have checked I have images locally by docker images

example $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE example latest af4bedf7589c 2 hours ago 1.15GB

but when I do: docker run example -it, it returns this: [FATAL tini (6)] exec -it failed: No such file or directory

— Reply to this email directly, view it on GitHub https://github.com/ploomber/soopervisor/pull/87#issuecomment-1167731477, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACYPJOJP35GOVK2U3PWQN2LVRHX5LANCNFSM5ZYUI6SA . You are receiving this because you were mentioned.Message ID: @.***>

[Wxl19980214]

docer run example:latest? It just returns nothing and not actually running the image.

[idomic]

Issue solved.

[edublancas]

so tests are passing now, I feel like we should tackle https://github.com/ploomber/soopervisor/issues/88 now to ensure this is working and also to have some recipe for future tests that involve the docker image.

thoughts? @idomic

[Wxl19980214]

@edublancas I think he meant local test are done. I am still figuring out a way to write automated tests.

[edublancas]

ok cool

[Wxl19980214]

@edublancas We have a PR for this branch already. I have commented in the text_export.py file.

[edublancas]

is this ready for review?

[Wxl19980214]

@edublancas Yes. I couldn't find where the changes were requested. I thought I resolved them all.

[idomic]

@Wxl19980214 Did you do some manual tests as well?

[Wxl19980214]

@idomic Yes I have tried with argo and kubernate. I can go ahead and check other backend as well

[idomic]

Please test from main with another backend let's say batch

[Wxl19980214]

Just tested. There is no issue.