Perform Query with single field which hits different values

[daramosch]

Hello!

I am trying to create an automated unmapped reclamation script. Which would be schedule as a cronjob every week and use the epoch value and volume time_stamps so if it surpass the time threshold it can be automatically reclaimed.

I was able to create an script that works on that matter, but is taking a lot of time to complete, I think I can speed up the time If I merge the Event search.

The logic of the script is the following: -Log into infinibox -Retrieve the Volumes that are unmapped -Search on the events for event.get_code()='HOST_UNMAPPED' which also contain the Volume name in the Description -If there is no hit, then search by event.get_code()='VOLUME_CREATED' (In case volume was never unmapped) -Retrieve User and Timestamp from event -Convert to timestamp to epoch, substract and verify if condition is met

I know there is a cleaner search if I do a loop like: "for events in system.events.find(code='HOST_UNMAPPED') is there a way to combine those 2 search for host unmapped and volume created into a single one?

Here is the output and the time values of completion. The Following Volumes are unmapped from: Infinibox_system

Volume Name: vol01 Owner: user@domain Unmapped_Timestamp: 2018-09-05 13:37:16 Candidate for reclaiming This Volume Has been Unmapped

Volume Name: testvol03 Owner: user@domain Unmapped_Timestamp: 2018-11-13 20:16:21 Has not reached idle threshold This Volume Has been Unmapped

Volume Name: testvol04 Owner: user@domain Unmapped_Timestamp: 2018-11-13 20:28:53 Has not reached idle threshold This Volume Has been Unmapped

real 5m15.482s user 0m38.981s sys 0m1.194s

Thanks a lot for your help!

[vmalloc]

A couple of questions:

1. How exactly are you fetching the events? There could be hidden performance costs in the way you retrieve them
2. How are you fetching all unmapped volumes?

Those are the two potential performance hits I could see in your flow

[daramosch]

Hi! Thanks for your quick reply. I was able to get the time down to seconds. I think the problem was within the for loop.

I was doing: import re for volume in system.volumes: if volume.is_mapped(): pass else: for event in system.events: if re.search(str(volume.get_name()),str(event.get_description())) and re.search(str(event.get_code()), 'HOST_UNMAPPED') else: if re.search(str(volume.get_name()),str(event.get_description())) and re.search(str(event.get_code()), 'VOLUME_CREATED')

---

It was solved by: for volume in system.volumes: if volume.is_mapped(): pass else: for event in system.events.find(code='HOST_UNMAPPED'): if re.search(str(volume.get_name()),str(event.get_description())): flag=1 if flag == 1: for event in system.events.find(code='VOLUME_CREATED'): if re.search(str(volume.get_name()),str(event.get_description())):

The Following Volumes are unmapped from: 'infinibox'

Volume Name: vol01 Owner: user@domain Unmapped_Timestamp: 2018-09-05 13:37:16 Candidate for reclaiming This Volume Has been Unmapped

Volume Name: testvol03 Owner: user@domain Creation_Timestamp: 2018-11-13 20:16:21 Has not reached idle threshold This Volume has never being mapped

Volume Name: testvol04 Owner: user@domain Unmapped_Timestamp: 2018-11-13 20:28:53 Has not reached idle threshold This Volume Has been Unmapped

Volume Name: volx Owner: user@domain Creation_Timestamp: 2018-11-15 12:28:28 Has not reached idle threshold This Volume has never being mapped

Volume Name: voly Owner: user@domain Creation_Timestamp: 2018-11-15 12:28:28 Has not reached idle threshold This Volume has never being mapped

real 0m12.838s user 0m1.378s sys 0m0.089s

[daramosch]

Thanks a lot for your reply, I was just wondering can I combine the .find function to look for events with code='HOST_UNMAPPED' and code='VOLUME_CREATED' ? I know I can include different fields such as description or id, but not sure if I can get a find to look for 2 different values on the same field on a single query

[vmalloc]

Ok, so to answer the first thing first:

import re

for volume in system.volumes:
    if volume.is_mapped():
        pass
    else:
        for event in system.events:
            if (re.search(str(volume.get_name()),
                          str(event.get_description()))
                and re.search(str(event.get_code()), 'HOST_UNMAPPED'):
                ...
            else:
                if re.search(str(volume.get_name()),str(event.get_description())) and re.search(str(event.get_code()), 'VOLUME_CREATED'):
                ...

1. You can search for volumes with a single query:

system.volumes.find(mapped=False)

2. You can filter events by code in advance:

   system.events.find(Q.code.in_(['HOST_UNMAPPED', 'VOLUME_CREATED'])

3. You don't need re. The volume's name is case-preserved, so just use in to check for string containment

4. (This is a bit of a pro tip because it's not very well documented) events have an affected_entity_id field you can search by, so to search for an event on volume v:

for event in system.events.find(affected_entity_id=v.id)

[daramosch]

Great! Thanks a lot for the explanation and the pro tip would be excellent in this case to search for the specific volume! I'll make sure to use it. I really appreciate your help in this matter!

[vmalloc]

My pleasure! Glad you got it sorted out.