search

dell / PyU4V

PyU4V is a Python module that simplifies interaction with the Unisphere for PowerMax REST API. It wraps REST calls with simple APIs that abstracts the HTTP request and response handling.
Apache License 2.0
41 stars 37 forks source link

Hangs when multiple calls made to failover_storage_group_srdf() with different SG names #191

Closed Sacheen123 closed 1 month ago

Sacheen123 commented 3 months ago

Thanks Paul for your reply in advance!!!

I have a following call in my code with different group_name & same rdf_number & opt as: replication.failover_storage_group_srdf(storage_group_id=self.group_name, srdf_group_number=self.rdf_number, failover_options=opt, _async=False)

I am calling above function at the same time for 6 storage groups. It is giving me exception: "Bad or unexpected response from the storage volume backend API: Error PUT symmetrix resource. The status code received is None and the message is None."

Also error message: ERROR:PyU4V.rest_requests:The PUT request to URL https://10.221.172.100:8443/univmax/restapi/92/replication/symmetrix/000297900701/storagegroup/SG_dl360g10-141_142_R6/rdf_group/101 timed-out, but may have been successful. Please check the array. Exception received: HTTPSConnectionPool(host='10.221.172.100', port=8443): Read timed out. (read timeout=120).

Looks like the API call hung caused the problem. I then used API to set timedout as below did not help me. self.conn = PyU4V.U4VConn(username=self.username, password=self.password, server_ip= self.server_ip, port=self.port, verify=self.verify,array_id=self.array_id) self.conn.set_requests_timeout(500)

Am I missing something? NOTE: We have both Gatekeeper & Unisphere available.

Thanks, Sacheen Birhade

rawstorage commented 3 months ago

Sacheen, you are issuing calls for 6 different groups at the same time, Try staggering the failovers, put a wait in doing them one after the other.

The error code is a generic one coming from the Unipshere server not from PyU4V, I suspect your Unipshere server is busy and replying back with the generic error. Another thing you could do is issue the calls asyncronously and check the job ids after, Check the examples in https://pyu4v.readthedocs.io/en/latest/programmers_guide_src/provisioning.html#a-synchronous-provisioning-creating-storage-for-a-host it's the same concept for all calls that support Asyncronous Execution, this is an ideal candidate for this type of operation. You won't be getting timeouts or overloading the server.

rawstorage commented 3 months ago

@Sacheen123 we have enabled the Discussions feature on GitHub for questions and Q&A, for non PyU4V issues and general discussions if you want to create discussions we can tackle on best effort basis.

Let me know if the Asynchronous calls resolved your issue, I'm confident that this would

Sacheen123 commented 3 months ago

Thanks Paul for your help. I have fixed this issue using exclusive lock through my python code. In my case multiple calls to the same API were through different thread, so file based exclusive lock helped me to send one request at a time. By that time all other threads were waiting. flow: take exclusive lock() all to api () release exclusive lock()

You were really helpful to me. Thanks. best wishes.

rawstorage commented 3 months ago

Nice trick 👌

Sent from Outlook for Androidhttps://aka.ms/AAb9ysg

Internal Use - Confidential

From: Sacheen123 @.> Sent: Friday, July 12, 2024 10:58:25 AM To: dell/PyU4V @.> Cc: Martin, Paul @.>; Assign @.> Subject: Re: [dell/PyU4V] Hangs when multiple calls made to failover_storage_group_srdf() with different SG names (Issue #191)

[EXTERNAL EMAIL]

Thanks Paul for your help. I have fixed this issue using exclusive lock through my python code. In my case multiple calls to the same API were through different thread, so file based exclusive lock helped me to send one request at a time. By that time all other threads were waiting. flow: take exclusive lock() all to api () release exclusive lock()

You were really helpful to me. Thanks. best wishes.

— Reply to this email directly, view it on GitHub [github.com]https://urldefense.com/v3/__https://github.com/dell/PyU4V/issues/191*issuecomment-2225235643__;Iw!!LpKI!nE7fM6-szl7eqsRCEKKsLR-oT3Cwbnt2sGeqEzrpd41-qDT3zo87garz8r7r8daGrrDNflr5oBxIUarObJmPXhyL$, or unsubscribe [github.com]https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AF7FWPIJFL4G64U7CDGVWODZL6SEDAVCNFSM6AAAAABKNP2DPWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRVGIZTKNRUGM__;!!LpKI!nE7fM6-szl7eqsRCEKKsLR-oT3Cwbnt2sGeqEzrpd41-qDT3zo87garz8r7r8daGrrDNflr5oBxIUarObOT6n2FT$. You are receiving this because you were assigned.Message ID: @.***>