Unable to use module - Githubissues

dralquinta commented 3 years ago

Hi!

After attempting to use ortu module, I'm consistently getting the following error:

[opc@dalquintdevhub lpg_config]$ terraform apply --var-file=myvars.tfvars

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # null_resource.from_route_table_update will be created
  + resource "null_resource" "from_route_table_update" {
      + id       = (known after apply)
      + triggers = {
          + "from_lpg" = "ocid1.localpeeringgateway.oc1.sa-santiago-1.aaaaaaaavp6pw6kadngkswzinshv7hy223rxc45vwkkd4zeuiuqwqyo5w62q"
        }
    }

  # null_resource.to_route_table_update will be created
  + resource "null_resource" "to_route_table_update" {
      + id       = (known after apply)
      + triggers = {
          + "to_lpg" = "ocid1.localpeeringgateway.oc1.sa-santiago-1.aaaaaaaaxeaw7qvnzckopr7ai6fa2r23rnnvotjzcrjepp6ur76i2jtad4ra"
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

After hitting yes, I get this:

null_resource.to_route_table_update: Creating...
null_resource.from_route_table_update: Creating...
null_resource.from_route_table_update: Provisioning with 'local-exec'...
null_resource.from_route_table_update (local-exec): Executing: ["/bin/sh" "-c" "ortu --rt-ocid ocid1.routetable.oc1.sa-santiago-1.aaaaaaaagjzhbjjipaehopoeozzhwo33n4pjhuokdj7qozf4japda7o2vmna --cidr ocid1.vcn.oc1.sa-santiago-1.amaaaaaatwfhi7ya2csaeycdh2vsxhuu2eyerufdnz6ydltkejyqpc6nqi5a --ne-ocid ocid1.localpeeringgateway.oc1.sa-santiago-1.aaaaaaaavp6pw6kadngkswzinshv7hy223rxc45vwkkd4zeuiuqwqyo5w62q"]
null_resource.to_route_table_update: Provisioning with 'local-exec'...
null_resource.to_route_table_update (local-exec): Executing: ["/bin/sh" "-c" "ortu --rt-ocid ocid1.routetable.oc1.sa-santiago-1.aaaaaaaav7xpvdfbo4j7eeqlgglmqoh5nu4egpls4cbou2idxubuj35353ia --cidr ocid1.vcn.oc1.sa-santiago-1.amaaaaaatwfhi7yasndc5vjypf2pgnfimyb2so756vmaimnozxlfpzu3uzhq --ne-ocid ocid1.localpeeringgateway.oc1.sa-santiago-1.aaaaaaaaxeaw7qvnzckopr7ai6fa2r23rnnvotjzcrjepp6ur76i2jtad4ra"]
null_resource.from_route_table_update (local-exec): Traceback (most recent call last):
null_resource.from_route_table_update (local-exec):   File "/home/opc/.local/bin/ortu", line 5, in <module>
null_resource.from_route_table_update (local-exec):     from ortu.main import main
null_resource.from_route_table_update (local-exec): ImportError: No module named ortu.main
null_resource.to_route_table_update (local-exec): Traceback (most recent call last):
null_resource.to_route_table_update (local-exec):   File "/home/opc/.local/bin/ortu", line 5, in <module>
null_resource.to_route_table_update (local-exec):     from ortu.main import main
null_resource.to_route_table_update (local-exec): ImportError: No module named ortu.main
╷
│ Error: local-exec provisioner error
│ 
│   with null_resource.to_route_table_update,
│   on lpgconfig.tf line 13, in resource "null_resource" "to_route_table_update":
│   13:   provisioner "local-exec" {
│ 
│ Error running command 'ortu --rt-ocid ocid1.routetable.oc1.sa-santiago-1.aaaaaaaav7xpvdfbo4j7eeqlgglmqoh5nu4egpls4cbou2idxubuj35353ia --cidr
│ ocid1.vcn.oc1.sa-santiago-1.amaaaaaatwfhi7yasndc5vjypf2pgnfimyb2so756vmaimnozxlfpzu3uzhq --ne-ocid ocid1.localpeeringgateway.oc1.sa-santiago-1.aaaaaaaaxeaw7qvnzckopr7ai6fa2r23rnnvotjzcrjepp6ur76i2jtad4ra': exit status
│ 1. Output: Traceback (most recent call last):
│   File "/home/opc/.local/bin/ortu", line 5, in <module>
│     from ortu.main import main
│ ImportError: No module named ortu.main
│ 
╵
╷
│ Error: local-exec provisioner error
│ 
│   with null_resource.from_route_table_update,
│   on lpgconfig.tf line 23, in resource "null_resource" "from_route_table_update":
│   23:   provisioner "local-exec" {
│ 
│ Error running command 'ortu --rt-ocid ocid1.routetable.oc1.sa-santiago-1.aaaaaaaagjzhbjjipaehopoeozzhwo33n4pjhuokdj7qozf4japda7o2vmna --cidr
│ ocid1.vcn.oc1.sa-santiago-1.amaaaaaatwfhi7ya2csaeycdh2vsxhuu2eyerufdnz6ydltkejyqpc6nqi5a --ne-ocid ocid1.localpeeringgateway.oc1.sa-santiago-1.aaaaaaaavp6pw6kadngkswzinshv7hy223rxc45vwkkd4zeuiuqwqyo5w62q': exit status
│ 1. Output: Traceback (most recent call last):
│   File "/home/opc/.local/bin/ortu", line 5, in <module>
│     from ortu.main import main
│ ImportError: No module named ortu.main

I've installed the module as instructed on documentation (pip install ortu)

I see installed module and indeed, I don't see it defined. This is what I get on .local binary:

#!/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from ortu.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

I've also tried with pip3, without good results unfortunately.

If executing ortu by it self, I get the same error as well:

[opc@dalquintdevhub lpg_config]$ ortu 
Traceback (most recent call last):
  File "/home/opc/.local/bin/ortu", line 5, in <module>
    from ortu.main import main
ImportError: No module named ortu.main

Is this something I'm doing wrong?

Thanks!

ITD27M01 commented 3 years ago

@dralquinta Hi, thank you for feedback,

I've never tested it with --user option because I'm using python virtualenv in CI, could you try the following:

pip3 install virtualenv --user
~/.local/bin/virtualenv -p python3 lpg_routes_config
source lpg_routes_config/bin/activate
pip3 install ortu
terraform apply --var-file=myvars.tfvars

Of course I'm going to investigate the issue with --user

dralquinta commented 3 years ago

Hi @ITD27M01

Thank you so much for your feedback!

I did some modifications on the code to support your suggestion and it seems to work just fine. For documentation this is what I did:

I extrapolated the logic you have on the sample code and transformed left to the concept "from" and right to the concept "to" (adding this for clarification :) )
in lpgconfig.tf I added the following logic on local exec, as I want to decouple the module execution from the bastion host it self:

lpgconfig.tf:

/* lpgroutes.tf 
Author: DALQUINT - denny.alquinta@oracle.com
Purpose: The following script adds the extra LPG Routes using project ortu implementation through null resources
Documentation: https://pypi.org/project/ortu/ 
Copyright (c) 2021, Oracle and/or its affiliates. All rights reserved. 
*/

resource "null_resource" "to_route_table_update" {
  triggers = {
    to_lpg_ocid     = local.to_lpg_ocid
    from_cidr_block = local.from_vcn_cidr
    to_rt_ocid      = local.to_rt_ocid
  }

  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]

    command = <<-EOT
    pip3 install virtualenv --user
    ~/.local/bin/virtualenv -p python3 lpg_routes_config
    source lpg_routes_config/bin/activate
    pip3 install ortu
    ortu --rt-ocid ${self.triggers.to_rt_ocid} --cidr ${self.triggers.from_cidr_block} --ne-ocid ${self.triggers.to_lpg_ocid}
    EOT
  }

  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]
    when        = destroy
    command     = <<-EOT
    pip3 install virtualenv --user
    ~/.local/bin/virtualenv -p python3 lpg_routes_config
    source lpg_routes_config/bin/activate
    pip3 install ortu
    ortu delete --rt-ocid ${self.triggers.to_rt_ocid} --cidr ${self.triggers.from_cidr_block} --ne-ocid ${self.triggers.to_lpg_ocid}
    EOT
  }

}

resource "null_resource" "from_route_table_update" {
  triggers = {
    from_lpg_ocid = local.from_lpg_ocid
    to_cidr_block = local.to_vcn_cidr
    from_rt_ocid  = local.from_rt_ocid
  }

  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]

    command = <<-EOT
    pip3 install virtualenv --user
    ~/.local/bin/virtualenv -p python3 lpg_routes_config
    source lpg_routes_config/bin/activate
    ortu --rt-ocid ${self.triggers.from_rt_ocid} --cidr ${self.triggers.to_cidr_block} --ne-ocid ${self.triggers.from_lpg_ocid}
    EOT
  }

  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]

    when    = destroy
    command = <<-EOT
    pip3 install virtualenv --user
    ~/.local/bin/virtualenv -p python3 lpg_routes_config
    source lpg_routes_config/bin/activate
    ortu delete --rt-ocid ${self.triggers.from_rt_ocid} --cidr ${self.triggers.to_cidr_block} --ne-ocid ${self.triggers.from_lpg_ocid}
    EOT
  }
}

and my datasource definition looks like this:

/* datasources.tf 
Author: DALQUINT - denny.alquinta@oracle.com
Purpose: The following script defines the lookup logic used in code to obtain pre-created resources in tenancy.
Copyright (c) 2021, Oracle and/or its affiliates. All rights reserved. 
*/

/********** Compartment and CF Accessors **********/
data "oci_identity_compartments" "FROMNWCOMPARTMENTS" {
  compartment_id            = var.tenancy_ocid
  compartment_id_in_subtree = true
  filter {
    name   = "name"
    values = [var.from_network_compartment_name]
  }
}

data "oci_core_vcns" "FROMVCN" {
  compartment_id = local.from_nw_compartment_ocid
  filter {
    name   = "display_name"
    values = [var.from_vcn_display_name]
  }
}

data "oci_core_route_tables" "FROMRT" {
  compartment_id = local.from_nw_compartment_ocid
  filter {
    name   = "display_name"
    values = [var.from_route_table_display_name]
  }
}

data "oci_core_local_peering_gateways" "FROMLPG" {

  compartment_id = local.from_nw_compartment_ocid
  vcn_id         = local.from_vcn_ocid
  filter {
    name   = "display_name"
    values = [var.from_lpg_display_name]
  }
}

data "oci_identity_compartments" "TONWCOMPARTMENTS" {
  compartment_id            = var.tenancy_ocid
  compartment_id_in_subtree = true
  filter {
    name   = "name"
    values = [var.to_network_compartment_name]
  }
}

data "oci_core_vcns" "TOVCN" {
  compartment_id = local.to_nw_compartment_ocid
  filter {
    name   = "display_name"
    values = [var.to_vcn_display_name]
  }
}

data "oci_core_route_tables" "TORT" {
  compartment_id = local.to_nw_compartment_ocid
  filter {
    name   = "display_name"
    values = [var.to_route_table_display_name]
  }
}

data "oci_core_local_peering_gateways" "TOLPG" {

  compartment_id = local.to_nw_compartment_ocid
  vcn_id         = local.to_vcn_ocid
  filter {
    name   = "display_name"
    values = [var.to_lpg_display_name]
  }
}

locals {

  /********** Compartment OCID Local Accessor **********/
  from_nw_compartment_ocid = length(data.oci_identity_compartments.FROMNWCOMPARTMENTS.compartments) > 0 ? lookup(data.oci_identity_compartments.FROMNWCOMPARTMENTS.compartments[0], "id") : null
  to_nw_compartment_ocid   = length(data.oci_identity_compartments.TONWCOMPARTMENTS.compartments) > 0 ? lookup(data.oci_identity_compartments.TONWCOMPARTMENTS.compartments[0], "id") : null

  /********** VCN OCID Local Accessor **********/
  from_vcn_ocid = lookup(data.oci_core_vcns.FROMVCN.virtual_networks[0], "id")
  to_vcn_ocid   = lookup(data.oci_core_vcns.TOVCN.virtual_networks[0], "id")

  /********** VCN CIDR Local Accessor **********/
  from_vcn_cidr = lookup(data.oci_core_vcns.FROMVCN.virtual_networks[0], "cidr_block")
  to_vcn_cidr   = lookup(data.oci_core_vcns.TOVCN.virtual_networks[0], "cidr_block")

  /********** Peered LPG Local Accessor **********/
  from_lpg_ocid = length(data.oci_core_local_peering_gateways.FROMLPG.local_peering_gateways) > 0 ? lookup(data.oci_core_local_peering_gateways.FROMLPG.local_peering_gateways[0], "id") : null
  to_lpg_ocid   = length(data.oci_core_local_peering_gateways.TOLPG.local_peering_gateways) > 0 ? lookup(data.oci_core_local_peering_gateways.TOLPG.local_peering_gateways[0], "id") : null

  /********** Associated Route Table **********/
  from_rt_ocid = lookup(data.oci_core_route_tables.FROMRT.route_tables[0], "id")
  to_rt_ocid   = lookup(data.oci_core_route_tables.TORT.route_tables[0], "id")
}

And my unitary tfvars looks like this:

######################################## ARTIFACT SPECIFIC VARIABLES ######################################
from_network_compartment_name = "TF_NETWORK_hub01comp"
from_vcn_display_name         = "HUB01_VCN_Terraform"
from_route_table_display_name = "HUB01_pvt_hub_rt"
from_lpg_display_name         = "HUB01_LPG01"

to_vcn_display_name         = "SPOKE01_VCN_Terraform"
to_network_compartment_name = "TF_NETWORK_spoke01comp"
to_lpg_display_name         = "SPOKE01_LPG01"
to_route_table_display_name = "SPOKE01_pvt_hub_rt"
######################################## ARTIFACT SPECIFIC VARIABLES ######################################

After these changes it works perfect!

In any case, it'll be a good idea to support -user from scratch, to avoid running the virtual env on local exec each time I run it (just to be tidy up, you know 👍 )

Other thing I noticed is that current code doesn't have support for Route Table Optional Description Entry. Any chance to add it? I can try to do it, but it'll probably take longer, so as you have it already on your head and in the way I see it, it's may boil down to reference the new variable on arguments.py and then adding the API call in oci_route_table.py.

Let me know if you want me to give it a shot with that updated change.

Thanks!

Denny.

ITD27M01 commented 3 years ago

@dralquinta Give me some time to investigate the description feature (after --user issue :)), I don't see any reference in SDK: https://oracle-cloud-infrastructure-python-sdk.readthedocs.io/en/latest/api/core/models/oci.core.models.UpdateRouteTableDetails.html#oci.core.models.UpdateRouteTableDetails

ITD27M01 commented 3 years ago

@dralquinta Looks like there are pip issues which describes your error message: https://github.com/pypa/pip/issues/5221 https://github.com/pypa/pip/issues/5599

The general advise of course is to use virtualenv for any pip packages and don't do things like pip install pip --upgrade

I've tested the native packages on Oracle Linux 8.4 and they work fine:

[cloud-user@tiunov-pip-test ~]$ cat /etc/oracle-release
Oracle Linux Server release 8.4

[cloud-user@tiunov-pip-test ~]$ sudo dnf install python3-pip
Last metadata expiration check: 0:08:03 ago on Fri 04 Jun 2021 06:41:58 AM UTC.
Dependencies resolved.
==============================================================================================================================================================================================================================================
 Package                                                Architecture                               Version                                                                        Repository                                             Size
==============================================================================================================================================================================================================================================
Installing:
 python3-pip                                            noarch                                     9.0.3-19.el8                                                                   ol8_appstream                                          20 k
Installing dependencies:
 python3-setuptools                                     noarch                                     39.2.0-6.el8                                                                   ol8_baseos_latest                                     163 k
 python36                                               x86_64                                     3.6.8-2.0.1.module+el8.4.0+20103+1849b5f9                                      ol8_appstream                                          19 k
Enabling module streams:
 python36                                                                                          3.6

Transaction Summary
==============================================================================================================================================================================================================================================
Install  3 Packages

Total download size: 202 k
Installed size: 466 k
Is this ok [y/N]: y
Downloading Packages:
(1/3): python36-3.6.8-2.0.1.module+el8.4.0+20103+1849b5f9.x86_64.rpm                                                                                                                                          100 kB/s |  19 kB     00:00
(2/3): python3-pip-9.0.3-19.el8.noarch.rpm                                                                                                                                                                     90 kB/s |  20 kB     00:00
(3/3): python3-setuptools-39.2.0-6.el8.noarch.rpm                                                                                                                                                             420 kB/s | 163 kB     00:00
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                         505 kB/s | 202 kB     00:00
Running transaction check
Running transaction
  Preparing        :                                                                                                                                                                                                                      1/1
  Installing       : python3-setuptools-39.2.0-6.el8.noarch                                                                                                                                                                               1/3
  Installing       : python3-pip-9.0.3-19.el8.noarch                                                                                                                                                                                      2/3
  Installing       : python36-3.6.8-2.0.1.module+el8.4.0+20103+1849b5f9.x86_64                                                                                                                                                            3/3
  Running scriptlet: python36-3.6.8-2.0.1.module+el8.4.0+20103+1849b5f9.x86_64                                                                                                                                                            3/3
  Verifying        : python3-setuptools-39.2.0-6.el8.noarch                                                                                                                                                                               1/3
  Verifying        : python3-pip-9.0.3-19.el8.noarch                                                                                                                                                                                      2/3
  Verifying        : python36-3.6.8-2.0.1.module+el8.4.0+20103+1849b5f9.x86_64                                                                                                                                                            3/3

Installed:
  python3-pip-9.0.3-19.el8.noarch                                     python3-setuptools-39.2.0-6.el8.noarch                                     python36-3.6.8-2.0.1.module+el8.4.0+20103+1849b5f9.x86_64

Complete!

[cloud-user@tiunov-pip-test ~]$ pip3 install ortu --user
Collecting ortu
  Downloading https://files.pythonhosted.org/packages/8d/34/ecd042708308fcf899e7970551b31921fd45f7e3719cc2b6cf82230101ec/ortu-0.5.0.tar.gz
Collecting oci>=2.26.0 (from ortu)
  Downloading https://files.pythonhosted.org/packages/50/e4/2eb53a050ea24da9a6ef8a8753582073c16abd98b1e5cecdf6371b390720/oci-2.39.0-py2.py3-none-any.whl (9.4MB)
    100% |████████████████████████████████| 9.4MB 153kB/s
Collecting pyOpenSSL<=19.1.0,>=17.5.0 (from oci>=2.26.0->ortu)
  Downloading https://files.pythonhosted.org/packages/9e/de/f8342b68fa9e981d348039954657bdf681b2ab93de27443be51865ffa310/pyOpenSSL-19.1.0-py2.py3-none-any.whl (53kB)
    100% |████████████████████████████████| 61kB 11.3MB/s
Collecting certifi (from oci>=2.26.0->ortu)
  Downloading https://files.pythonhosted.org/packages/05/1b/0a0dece0e8aa492a6ec9e4ad2fe366b511558cdc73fd3abc82ba7348e875/certifi-2021.5.30-py2.py3-none-any.whl (145kB)
    100% |████████████████████████████████| 153kB 5.5MB/s
Requirement already satisfied: pytz>=2016.10 in /usr/lib/python3.6/site-packages (from oci>=2.26.0->ortu)
Collecting configparser==4.0.2 (from oci>=2.26.0->ortu)
  Downloading https://files.pythonhosted.org/packages/7a/2a/95ed0501cf5d8709490b1d3a3f9b5cf340da6c433f896bbe9ce08dbe6785/configparser-4.0.2-py2.py3-none-any.whl
Requirement already satisfied: python-dateutil<3.0.0,>=2.5.3 in /usr/lib/python3.6/site-packages (from oci>=2.26.0->ortu)
Collecting cryptography==3.3.2 (from oci>=2.26.0->ortu)
  Downloading https://files.pythonhosted.org/packages/32/48/ec2a3e98d8b61d2c65e4c6905aad370049c4bd4a6b1b1d78f8983d38effe/cryptography-3.3.2-cp36-abi3-manylinux1_x86_64.whl (2.7MB)
    100% |████████████████████████████████| 2.7MB 467kB/s
Requirement already satisfied: six>=1.5.2 in /usr/lib/python3.6/site-packages (from pyOpenSSL<=19.1.0,>=17.5.0->oci>=2.26.0->ortu)
Collecting cffi>=1.12 (from cryptography==3.3.2->oci>=2.26.0->ortu)
  Downloading https://files.pythonhosted.org/packages/2b/cf/9a3b04e57191a970836aeaa8b2075574f02fbdb65d6368457a2f13213e7f/cffi-1.14.5-cp36-cp36m-manylinux1_x86_64.whl (401kB)
    100% |████████████████████████████████| 409kB 3.1MB/s
Requirement already satisfied: pycparser in /usr/lib/python3.6/site-packages (from cffi>=1.12->cryptography==3.3.2->oci>=2.26.0->ortu)
Installing collected packages: cffi, cryptography, pyOpenSSL, certifi, configparser, oci, ortu
  Running setup.py install for ortu ... done
Successfully installed certifi-2021.5.30 cffi-1.14.5 configparser-4.0.2 cryptography-3.3.2 oci-2.39.0 ortu-0.5.0 pyOpenSSL-19.1.0

[cloud-user@tiunov-pip-test ~]$ ortu --version
usage: ortu [-h] [--rt-ocid RT_OCID] [--cidr CIDR] [--ne-ocid NE_OCID]
            [--debug] [--dry-run]
            [{create,delete}]
ortu: error: unrecognized arguments: --version
[cloud-user@tiunov-pip-test ~]$ ortu --dry-run
Something went wrong: Route table OCID is required. Try to --debug it

ITD27M01 commented 3 years ago

@dralquinta Could you confirm that I can close this?

dralquinta commented 3 years ago

Yup. I think it's in conditions to be closed at this point. Thanks!

ITD27M01 / oci-route-table-updater

Unable to use module #8

lpgconfig.tf: