search

oracle / oci-python-sdk

Oracle Cloud Infrastructure SDK for Python
https://cloud.oracle.com/cloud-infrastructure
Other
386 stars 277 forks source link

showoci.py script error - please help! #263

Closed goldiesharma closed 4 years ago

goldiesharma commented 4 years ago

[root@ocivnsysoats211 showoci]# python3 showoci.py -ip -ani

############################################################

showoci.py

############################################################ Author : Adi Zohar Machine : ocivnsysoats211 (x86_64) Python Version : 3.6.8 Authentication : Instance Principals Date/Time : 2020-06-17 15:43:32 Comand Line : -ip -ani Showoci Version : 20.06.16 OCI SDK Version : 2.16.0

############################################################

Load OCI data to Memory

############################################################ Guide: '.' Compartment, '+' VCN, '-' Subnets, 'a' - auth/notfound

Identity... --> Tenancy <-- (1) - 00:00:00 --> Compartments <-- Error in ShowOCIService:load_identity_main: : Error in load_identity_compartments: ("Error in add_tenant_compartment: ({'opc-request-id': 'D61C5CC19EDC4B97BEE4D0C5D7D44C36/8D6379B64D6924E92793371524A9CDCD/1D6E183055929AF8D64413D64B36865E', 'code': 'NotAuthorizedOrNotFound', 'message': 'Authorization failed or requested resource not found', 'status': 404},)",) Announcements... --> Announcement Items <-- a. (0) - 00:00:00

##############################

Region us-ashburn-1

############################## Identity... ^C--> Availability Domains <-- Traceback (most recent call last): File "showoci.py", line 451, in execute_extract() File "showoci.py", line 137, in execute_extract if not data.load_service_data(): File "/home/gsharma/oci-python-sdk/examples/showoci/showoci_data.py", line 56, in load_service_data return self.service.load_service_data() File "/home/gsharma/oci-python-sdk/examples/showoci/showoci_service.py", line 498, in load_service_data return self.load_data_main() File "/home/gsharma/oci-python-sdk/examples/showoci/showoci_service.py", line 805, in load_data_main self.load_oci_region_data(region_name) File "/home/gsharma/oci-python-sdk/examples/showoci/showoci_service.py", line 828, in load_oci_region_data self.__load_identity_availability_domain(region_name) File "/home/gsharma/oci-python-sdk/examples/showoci/showoci_service.py", line 1645, in __load_identity_availability_domain availability_domains = identity.list_availability_domains(self.get_tenancy_id()).data File "/usr/lib/python3.6/site-packages/oci/identity/identity_client.py", line 5295, in list_availability_domains response_type="list[AvailabilityDomain]") File "/usr/lib/python3.6/site-packages/oci/base_client.py", line 248, in call_api return self.request(request) File "/usr/lib/python3.6/site-packages/oci/base_client.py", line 348, in request timeout=self.timeout) File "/usr/lib/python3.6/site-packages/oci/_vendor/requests/sessions.py", line 538, in request resp = self.send(prep, send_kwargs) File "/usr/lib/python3.6/site-packages/oci/_vendor/requests/sessions.py", line 651, in send r = adapter.send(request, kwargs) File "/usr/lib/python3.6/site-packages/oci/_vendor/requests/adapters.py", line 457, in send enforce_content_length=True File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/connectionpool.py", line 605, in urlopen chunked=chunked) File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/connectionpool.py", line 348, in _make_request self._validate_conn(conn) File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/connectionpool.py", line 844, in _validate_conn conn.connect() File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/connection.py", line 306, in connect conn = self._new_conn() File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/connection.py", line 164, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/util/connection.py", line 75, in create_connection

adizohar commented 4 years ago

Hi , you probably didn't create policy or dynamic group to allow the compute to run the script Can you please share the dynamic group and the policy ? Did you follow the step by step instructions - https://github.com/oracle/oci-python-sdk/blob/master/examples/showoci/step_by_step_installation.md

goldiesharma commented 4 years ago

i have installed oci cli on my running linux instance, and downloaded showoci script. My .config file has all priviledges to run any commands i need, do i still need to create dymanic group?

I have also tried below command that failed too... Appreciate your help!

[root@ocivnsysoats211 showoci]# python3 showoci.py -t ~/.oci/config -ani

[root@ocivnsysoats211 showoci]# cat /root/.oci/config [DEFAULT] user=ocid1.user.oc1..aaabfze7exhgzjoru6dgv4qlcaeeyescy5ay6pipsq fingerprint=12:70:9c:30::e9:e6:6b:eb:1b:30:82:59 key_file=~/.oci/oci_api_key.pem tenancy=ocid1.tenancy.oc1..aaaaaaaa4ms4g57xldvbl7mxrqogbw6yxeyb4zdnpfuuolpadyrh7gf2nrma region=us-phoenix-1

[root@ocivnsysoats211 showoci]# pwd /home/gsharma/oci-python-sdk/examples/showoci [root@ocivnsysoats211 showoci]# ls -ltr total 840 -rw-r--r-- 1 root root 2791 Jun 15 17:14 step_by_step_installation.md -rwxr-xr-x 1 root root 60058 Jun 15 17:14 showoci_to_se.py -rwxr-xr-x 1 root root 382105 Jun 15 17:14 showoci_service.py -rwxr-xr-x 1 root root 144719 Jun 15 17:14 showoci_output.py drwxr-xr-x 2 root root 4096 Jun 15 17:14 img -rwxr-xr-x 1 root root 12996 Jun 17 15:40 CHANGELOG.rst -rwxr-xr-x 1 root root 84756 Jun 17 15:40 README.md -rwxr-xr-x 1 root root 2667 Jun 17 15:40 run_daily_report.sh -rwxr-xr-x 1 root root 19379 Jun 17 15:40 showoci.py -rwxr-xr-x 1 root root 129403 Jun 17 15:40 showoci_data.py drwxr-xr-x 2 root root 116 Jun 17 15:43 pycache [root@ocivnsysoats211 showoci]#

goldiesharma commented 4 years ago

Hi Adi,

looks like after creating dynamic group , script started moving but it failed below, also would like to know how do i get csv report.

[root@ocivnsysoats211 showoci]# python3 showoci.py -ip -ani

############################################################

showoci.py

############################################################ Author : Adi Zohar Machine : ocivnsysoats211 (x86_64) Python Version : 3.6.8 Authentication : Instance Principals Date/Time : 2020-06-17 17:37:21 Comand Line : -ip -ani Showoci Version : 20.06.16 OCI SDK Version : 2.16.0

############################################################

Load OCI data to Memory

############################################################ Guide: '.' Compartment, '+' VCN, '-' Subnets, 'a' - auth/notfound

Identity... --> Tenancy <-- (1) - 00:00:00 --> Compartments <-- (46) - 00:00:00

Announcements... --> Announcement Items <-- . (5) - 00:00:01

##############################

Region us-ashburn-1

############################## Identity... --> Availability Domains <-- Error in ShowOCIService:__load_data_main: : HTTPSConnectionPool(host='identity.us-ashburn-1.oraclecloud.com', port=443): Max retries exceeded with url: /20160918/availabilityDomains?compartmentId=ocid1.tenancy.oc1..aaaaaaaa4ms4g57xldvbl7mxrqogbw6yxeyb4zdnpfuuolpadyrh7gf2nrma (Caused by NewConnectionError('<oci._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f590c4376a0>: Failed to establish a new connection: [Errno 110] Connection timed out',)) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/connection.py", line 164, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/util/connection.py", line 85, in create_connection raise err File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/util/connection.py", line 75, in create_connection sock.connect(sa) TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/connectionpool.py", line 605, in urlopen chunked=chunked) File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/connectionpool.py", line 348, in _make_request self._validate_conn(conn) File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/connectionpool.py", line 844, in _validate_conn conn.connect() File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/connection.py", line 306, in connect conn = self._new_conn() File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/connection.py", line 173, in _new_conn self, "Failed to establish a new connection: %s" % e) oci._vendor.urllib3.exceptions.NewConnectionError: <oci._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f590c4376a0>: Failed to establish a new connection: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/oci/_vendor/requests/adapters.py", line 457, in send enforce_content_length=True File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/connectionpool.py", line 643, in urlopen _stacktrace=sys.exc_info()[2]) File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/util/retry.py", line 403, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) oci._vendor.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='identity.us-ashburn-1.oraclecloud.com', port=443): Max retries exceeded with url: /20160918/availabilityDomains?compartmentId=ocid1.tenancy.oc1..aaaaaaaa4ms4g57xldvbl7mxrqogbw6yxeyb4zdnpfuuolpadyrh7gf2nrma (Caused by NewConnectionError('<oci._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f590c4376a0>: Failed to establish a new connection: [Errno 110] Connection timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/oci/base_client.py", line 348, in request timeout=self.timeout) File "/usr/lib/python3.6/site-packages/oci/_vendor/requests/sessions.py", line 538, in request resp = self.send(prep, send_kwargs) File "/usr/lib/python3.6/site-packages/oci/_vendor/requests/sessions.py", line 651, in send r = adapter.send(request, kwargs) File "/usr/lib/python3.6/site-packages/oci/_vendor/requests/adapters.py", line 524, in send raise ConnectionError(e, request=request) oci._vendor.requests.exceptions.ConnectionError: HTTPSConnectionPool(host='identity.us-ashburn-1.oraclecloud.com', port=443): Max retries exceeded with url: /20160918/availabilityDomains?compartmentId=ocid1.tenancy.oc1..aaaaaaaa4ms4g57xldvbl7mxrqogbw6yxeyb4zdnpfuuolpadyrh7gf2nrma (Caused by NewConnectionError('<oci._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f590c4376a0>: Failed to establish a new connection: [Errno 110] Connection timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "showoci.py", line 451, in execute_extract() File "showoci.py", line 137, in execute_extract if not data.load_service_data(): File "/home/gsharma/oci-python-sdk/examples/showoci/showoci_data.py", line 56, in load_service_data return self.service.load_service_data() File "/home/gsharma/oci-python-sdk/examples/showoci/showoci_service.py", line 498, in load_service_data return self.load_data_main() File "/home/gsharma/oci-python-sdk/examples/showoci/showoci_service.py", line 805, in load_data_main self.load_oci_region_data(region_name) File "/home/gsharma/oci-python-sdk/examples/showoci/showoci_service.py", line 828, in load_oci_region_data self.__load_identity_availability_domain(region_name) File "/home/gsharma/oci-python-sdk/examples/showoci/showoci_service.py", line 1645, in __load_identity_availability_domain availability_domains = identity.list_availability_domains(self.get_tenancy_id()).data File "/usr/lib/python3.6/site-packages/oci/identity/identity_client.py", line 5295, in list_availability_domains response_type="list[AvailabilityDomain]") File "/usr/lib/python3.6/site-packages/oci/base_client.py", line 248, in call_api return self.request(request) File "/usr/lib/python3.6/site-packages/oci/base_client.py", line 357, in request raise exceptions.RequestException(e) oci.exceptions.RequestException: HTTPSConnectionPool(host='identity.us-ashburn-1.oraclecloud.com', port=443): Max retries exceeded with url: /20160918/availabilityDomains?compartmentId=ocid1.tenancy.oc1..aaaaaaaa4ms4g57xldvbl7mxrqogbw6yxeyb4zdnpfuuolpadyrh7gf2nrma (Caused by NewConnectionError('<oci._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f590c4376a0>: Failed to establish a new connection: [Errno 110] Connection timed out',)) [root@ocivnsysoats211 showoci]#

adizohar commented 4 years ago

if you use user authentication, please run without the -ip: python3 showoci.py -ani Please let me know how does it go For your info Adi

goldiesharma commented 4 years ago

root@ocivnsysoats211 showoci]# python3 showoci.py -ani

############################################################

showoci.py

############################################################ Author : Adi Zohar Machine : ocivnsysoats211 (x86_64) Python Version : 3.6.8 Authentication : Config File Config File : ~/.oci/config Config Profile : DEFAULT Date/Time : 2020-06-17 17:46:18 Comand Line : -ani Showoci Version : 20.06.16 OCI SDK Version : 2.16.0

############################################################

Load OCI data to Memory

############################################################ Guide: '.' Compartment, '+' VCN, '-' Subnets, 'a' - auth/notfound

Identity... --> Tenancy <-- (1) - 00:00:01 --> Compartments <-- (46) - 00:00:00

Announcements... --> Announcement Items <-- . (5) - 00:00:01

##############################

Region us-ashburn-1

############################## Identity...

Waiting here for a long time i guess i will get the same error<<<<<<<<<<

goldiesharma commented 4 years ago

showocidynamicgroup showocipol

adizohar commented 4 years ago

The run looks better, you have 46 compartments can take time Please add below to get feedback before you run export PYTHONUNBUFFERED=TRUE

goldiesharma commented 4 years ago

It err'd out...

[root@ocivnsysoats211 showoci]# python3 showoci.py -ani

############################################################

showoci.py

############################################################ Author : Adi Zohar Machine : ocivnsysoats211 (x86_64) Python Version : 3.6.8 Authentication : Config File Config File : ~/.oci/config Config Profile : DEFAULT Date/Time : 2020-06-17 17:46:18 Comand Line : -ani Showoci Version : 20.06.16 OCI SDK Version : 2.16.0

############################################################

Load OCI data to Memory

############################################################ Guide: '.' Compartment, '+' VCN, '-' Subnets, 'a' - auth/notfound

Identity... --> Tenancy <-- (1) - 00:00:01 --> Compartments <-- (46) - 00:00:00

Announcements... --> Announcement Items <-- . (5) - 00:00:01

##############################

Region us-ashburn-1

############################## Identity... --> Availability Domains <-- Error in ShowOCIService:__load_data_main: : HTTPSConnectionPool(host='identity.us-ashburn-1.oraclecloud.com', port=443): Max retries exceeded with url: /20160918/availabilityDomains?compartmentId=ocid1.tenancy.oc1..aaaaaaaa4ms4g57xldvbl7mxrqogbw6yxeyb4zdnpfuuolpadyrh7gf2nrma (Caused by NewConnectionError('<oci._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7efe1a345e80>: Failed to establish a new connection: [Errno 110] Connection timed out',)) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/connection.py", line 164, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/util/connection.py", line 85, in create_connection raise err File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/util/connection.py", line 75, in create_connection sock.connect(sa) TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/connectionpool.py", line 605, in urlopen chunked=chunked) File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/connectionpool.py", line 348, in _make_request self._validate_conn(conn) File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/connectionpool.py", line 844, in _validate_conn conn.connect() File "/usr/lib/python3.6/site-packages/oci/_vendo

Running it again

[root@ocivnsysoats211 showoci]# export PYTHONUNBUFFERED=TRUE [root@ocivnsysoats211 showoci]# python3 showoci.py -ani

############################################################

showoci.py

############################################################ Author : Adi Zohar Machine : ocivnsysoats211 (x86_64) Python Version : 3.6.8 Authentication : Config File Config File : ~/.oci/config Config Profile : DEFAULT Date/Time : 2020-06-17 17:56:01 Comand Line : -ani Showoci Version : 20.06.16 OCI SDK Version : 2.16.0

############################################################

Load OCI data to Memory

############################################################ Guide: '.' Compartment, '+' VCN, '-' Subnets, 'a' - auth/notfound

Identity... --> Tenancy <-- (1) - 00:00:00 --> Compartments <-- (46) - 00:00:00

Announcements... --> Announcement Items <-- . (5) - 00:00:01

##############################

Region us-ashburn-1

############################## Identity...

adizohar commented 4 years ago

Please let me know if this work Thank you

goldiesharma commented 4 years ago

failed again...

##############################

Region us-ashburn-1

############################## Identity... --> Availability Domains <-- Error in ShowOCIService:__load_data_main: : HTTPSConnectionPool(host='identity.us-ashburn-1.oraclecloud.com', port=443): Max retries exceeded with url: /20160918/availabilityDomains?compartmentId=ocid1.tenancy.oc1..aaaaaaaa4ms4g57xldvbl7mxrqogbw6yxeyb4zdnpfuuolpadyrh7gf2nrma (Caused by NewConnectionError('<oci._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f3365699ef0>: Failed to establish a new connection: [Errno 110] Connection timed out',)) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/oci/_vendor/urllib3/connection.py", line 164, in _new_conn

goldiesharma commented 4 years ago

I hope this is not a firewall issue? it says connection timeout! Could this be a permission issue?

adizohar commented 4 years ago

Do you have NAT GW or Internet GW on the VCN of your compute ? it looks like you get timeout accessing the basic API (availability domains) Can you run only network ? or one compartment showoci.py -n ? or showoci.py -n -cp COMPARTMENT_NAME

goldiesharma commented 4 years ago

[root@ocivnsysoats211 showoci]# python3 showoci.py -n -cp EASO-P-1-OATS-CP-21

############################################################

showoci.py

############################################################ Author : Adi Zohar Machine : ocivnsysoats211 (x86_64) Python Version : 3.6.8 Authentication : Config File Config File : ~/.oci/config Config Profile : DEFAULT Date/Time : 2020-06-17 18:08:53 Comand Line : -n -cp EASO-P-1-OATS-CP-21 Showoci Version : 20.06.16 OCI SDK Version : 2.16.0

############################################################

Load OCI data to Memory

############################################################ Guide: '.' Compartment, '+' VCN, '-' Subnets, 'a' - auth/notfound Filtered by Compartment like EASO-P-1-OATS-CP-21

Identity... --> Tenancy <-- (1) - 00:00:00 --> Compartments <-- (1) - 00:00:00

##############################

Region us-ashburn-1

############################## Identity...

#############HANGING HERE I GUESS SAME ISSUE ############

goldiesharma commented 4 years ago

my ingress/egress is wide opened 0.0.0.0 for this instance.

goldiesharma commented 4 years ago

can i run some telnet/curl command to check if any firewall issues? could this be a permission issue accessing domain, what do i need to check on permission?

adizohar commented 4 years ago

I am happy to work with you tomorrow to solve it

goldiesharma commented 4 years ago

what command is running on backend, i have oci-cli tool which i can run manually and check if oci-cli commands works?

goldiesharma commented 4 years ago

ok no worries, please ping me when you have sometime tomorrow. Appreciate your help!

adizohar commented 4 years ago

Can you run: oci iam availability-domain list

adizohar commented 4 years ago

Ping me at morning and I will check for you the config

goldiesharma commented 4 years ago

[root@ocivnsysoats211 showoci]# oci iam availability-domain list { "data": [ { "compartment-id": "ocid1.tenancy.oc1..aaaaaaaa4ms4g57xldvbl7mxrqogbw6yxeyb4zdnpfuuolpadyrh7gf2nrma", "id": "ocid1.availabilitydomain.oc1..aaaaaaaaupx7ga46w4xbb3fnkoh7rtyumacyb3ek6bw7bt3qff3mgt3mbrlq", "name": "ialc:PHX-AD-1" }, { "compartment-id": "ocid1.tenancy.oc1..aaaaaaaa4ms4g57xldvbl7mxrqogbw6yxeyb4zdnpfuuolpadyrh7gf2nrma", "id": "ocid1.availabilitydomain.oc1..aaaaaaaakemzo4ywaynz573sk4jpvwuqi7vmcuxajxfv4jjj2pthez3j2hbq", "name": "ialc:PHX-AD-2" }, { "compartment-id": "ocid1.tenancy.oc1..aaaaaaaa4ms4g57xldvbl7mxrqogbw6yxeyb4zdnpfuuolpadyrh7gf2nrma", "id": "ocid1.availabilitydomain.oc1..aaaaaaaac7uydq54rcymamzauv5pd23qk2g2vqlrfu5jd3y6i6qxpvvzkf7a", "name": "ialc:PHX-AD-3" } ] } [root@ocivnsysoats211 showoci]#

goldiesharma commented 4 years ago

doesn't look like its firewall issue...

[root@ocivnsysoats211 showoci]# curl -s --head https://iaas.us-phoenix-1.oraclecloud.com | grep Date Date: Thu, 18 Jun 2020 02:20:42 GMT

adizohar commented 4 years ago

Please run it on Ashburn, this is where it hung Change your ~/.oci/config file to us-ashburn-1 region and run again and oci cli command

goldiesharma commented 4 years ago

Same Error even after changing the region.

############## [root@ocivnsysoats211 showoci]# cat ~/.oci/config |grep region region=us-ashburn-1

############################################################

Load OCI data to Memory

############################################################ Guide: '.' Compartment, '+' VCN, '-' Subnets, 'a' - auth/notfound Filtered by Compartment like EASO-P-1-OATS-CP-21

Identity... ^C--> Tenancy <-- Traceback (most recent call last): File "showoci.py", line 451, in execute_extract() File "showoci.py", line 137, in execute_extract if not data.load_service_data(): File "/home/gsharma/oci-python-sdk/examples/showoci/showoci_data.py", line 56, in load_service_data

goldiesharma commented 4 years ago

spent long time to figure out the issue, basically it was a region issue.

i figured out how to input specific region name, and its working perfectly fine now..

Thank you so much Adi for implementing such amazing script!

python3 showoci.py -ip -ani -rg us-phoenix-1 (This is what worked)

adizohar commented 4 years ago

If all worked, do you still need to have the ticket opened ?

goldiesharma commented 4 years ago

Yep all worked... we can close the ticket! one last question if i need to schedule the script to run daily/weekly any thoughts on this?

jodoglevy commented 4 years ago

Closing per comments above