search

lsst-uk / somerville-operations

User issue reporting and tracking for the Somerville Cloud
0 stars 0 forks source link

review ceph-ssd quotas #190

Closed GregBlow closed 2 weeks ago

GregBlow commented 2 weeks ago

review ceph-ssd quotas t oensure not oversubscribed.

GregBlow commented 2 weeks ago 
(openstack-config) [stack@sv-admin-0 openstack-config]$ openstack project list -c Name -f csv | xargs -tn 1 openstack quota
show | grep gigabytes_ceph-ssd
openstack quota show Name
No project with a name or ID of 'Name' exists.
openstack quota show 5b5102968e5347ad98676ea42b6519df-16f3d336-a7b9-44be-99c1-e1ad98c
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show service
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show tempest-ServersTestJSON-606998978
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show tempest-AttachVolumeTestJSON-1482633129
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show tempest-TestVersions-1963405460
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show RSP-test
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show d3540d2620c147f1a528bcd2bb5c4926-21ef8234-cf58-42ad-b671-3c6d579
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show tempest-SecurityGroupsTestJSON-1409216901
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show tempest-ListServersNegativeTestJSON-1690772054
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show af29bdf3eb4d4467a855f54a0441d9ab-57e6b468-3507-4ddd-984a-3997cd4
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show RSP
| gigabytes_ceph-ssd    | 16000
                                                                                         |
openstack quota show VDFS
| gigabytes_ceph-ssd    | 1000
                                                                                         |
openstack quota show sandbox
| gigabytes_ceph-ssd    | 1000
                                                                                         |
openstack quota show 'VDFS VVV'
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show tempest-QuotasTestJSON-1923675267
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show d3540d2620c147f1a528bcd2bb5c4926-f5320729-f005-48aa-a4eb-e59c2dc
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show iris-portal-demo
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show d3540d2620c147f1a528bcd2bb5c4926-8695fa99-550e-4136-a336-c0a10b3
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show tempest-FlavorsV2TestJSON-272561241
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show tempest-ServersTestJSON-1855721018
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show tempest-InstanceActionsTestJSON-66907409
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show 5b5102968e5347ad98676ea42b6519df-bd673cbc-19cd-4819-9422-7a5e5d5
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show af29bdf3eb4d4467a855f54a0441d9ab-9d049df2-b2f7-4e50-a7a7-2402988
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show 'Acceptance testing'
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show Qserv
| gigabytes_ceph-ssd    | 80000
                                                                                         |
openstack quota show Systems
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show af29bdf3eb4d4467a855f54a0441d9ab-27abac1d-9604-4653-be36-915f8a8
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show d3540d2620c147f1a528bcd2bb5c4926-e6b122ea-1e16-4bb7-8262-28b7ac7
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show tempest-ServersTestJSON-1076937420
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show Perfsonar
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show tempest-ListServerFiltersTestJSON-719602640
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show tempest-MultipleCreateTestJSON-783109946
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show test
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show magnum-mgmt
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show Somerville-Gaia-Jade
| gigabytes_ceph-ssd    | 1000
                                                                                         |
openstack quota show tempest-DeleteServersTestJSON-93301536
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show tempest-ServerActionsTestJSON-1181309718
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show Lasair
| gigabytes_ceph-ssd    | 32000
                                                                                         |
openstack quota show TiDES
| gigabytes_ceph-ssd    | 1000
                                                                                         |
openstack quota show Adler
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show tempest-ServersTestJSON-295113466
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show af29bdf3eb4d4467a855f54a0441d9ab-8ce826ea-cf31-47e7-b798-b6a131c
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show tempest-ImagesOneServerTestJSON-2074285245
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show tempest-ServersTestManualDisk-1847089014
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show capi
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show tempest-TokensV3Test-1611948511
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show d3540d2620c147f1a528bcd2bb5c4926-31a5374b-5bef-4b5c-b65a-d9cc916
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show tempest-ServersNegativeTestJSON-1030576039
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show tempest-SecurityGroupsNegativeTestJSON-1580739083
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show tempest-ServersTestJSON-732424071
| gigabytes_ceph-ssd    | -1
                                                                                         |
openstack quota show admin
| gigabytes_ceph-ssd    | 1000
                                                                                         |
GregBlow commented 2 weeks ago

current qserv usage is 17TB (35TB prod10, 21TB prod5)

GregBlow commented 2 weeks ago

(openstack-config) [stack@sv-admin-0 openstack-config]$ openstack quota set --gigabytes 17000 --volume-type ceph-ssd Qserv

GregBlow commented 2 weeks ago

apart from qserv and lasair, the following projects have ceph-ssd quota: VDFS: 1TB sandbox: 1TB Gaia: 1TB TiDES: 1TB admin: 1TB

GregBlow commented 2 weeks ago 
(openstack-config) [stack@sv-admin-0 openstack-config]$ openstack quota set --gigabytes -1 --volume-type ceph-ssd VDFS
(openstack-config) [stack@sv-admin-0 openstack-config]$ openstack quota set --gigabytes -1 --volume-type ceph-ssd TiDES
(openstack-config) [stack@sv-admin-0 openstack-config]$ openstack quota set --gigabytes -1 --volume-type ceph-ssd Somerville-Gaia-Jade
GregBlow commented 2 weeks ago 
(openstack-config) [stack@sv-admin-0 openstack-config]$ openstack project list -c Name -f csv | xargs -tn 1 openstack quota
show | grep ceph-ssd
openstack quota show Name
No project with a name or ID of 'Name' exists.
openstack quota show 5b5102968e5347ad98676ea42b6519df-16f3d336-a7b9-44be-99c1-e1ad98c
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show service
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-ServersTestJSON-606998978
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-AttachVolumeTestJSON-1482633129
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-TestVersions-1963405460
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show RSP-test
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show d3540d2620c147f1a528bcd2bb5c4926-21ef8234-cf58-42ad-b671-3c6d579
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-SecurityGroupsTestJSON-1409216901
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-ListServersNegativeTestJSON-1690772054
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show af29bdf3eb4d4467a855f54a0441d9ab-57e6b468-3507-4ddd-984a-3997cd4
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show RSP
| gigabytes_ceph-ssd    | 16000
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show VDFS
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show sandbox
| gigabytes_ceph-ssd    | 1000
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show 'VDFS VVV'
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-QuotasTestJSON-1923675267
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show d3540d2620c147f1a528bcd2bb5c4926-f5320729-f005-48aa-a4eb-e59c2dc
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show iris-portal-demo
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show d3540d2620c147f1a528bcd2bb5c4926-8695fa99-550e-4136-a336-c0a10b3
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-FlavorsV2TestJSON-272561241
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-ServersTestJSON-1855721018
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-InstanceActionsTestJSON-66907409
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show 5b5102968e5347ad98676ea42b6519df-bd673cbc-19cd-4819-9422-7a5e5d5
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show af29bdf3eb4d4467a855f54a0441d9ab-9d049df2-b2f7-4e50-a7a7-2402988
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show 'Acceptance testing'
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show Qserv
| gigabytes_ceph-ssd    | 17000
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show Systems
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show af29bdf3eb4d4467a855f54a0441d9ab-27abac1d-9604-4653-be36-915f8a8
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show d3540d2620c147f1a528bcd2bb5c4926-e6b122ea-1e16-4bb7-8262-28b7ac7
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-ServersTestJSON-1076937420
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show Perfsonar
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-ListServerFiltersTestJSON-719602640
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-MultipleCreateTestJSON-783109946
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show test
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show magnum-mgmt
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show Somerville-Gaia-Jade
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-DeleteServersTestJSON-93301536
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-ServerActionsTestJSON-1181309718
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show Lasair
| gigabytes_ceph-ssd    | 32000
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show TiDES
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show Adler
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-ServersTestJSON-295113466
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show af29bdf3eb4d4467a855f54a0441d9ab-8ce826ea-cf31-47e7-b798-b6a131c
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-ImagesOneServerTestJSON-2074285245
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-ServersTestManualDisk-1847089014
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show capi
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-TokensV3Test-1611948511
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show d3540d2620c147f1a528bcd2bb5c4926-31a5374b-5bef-4b5c-b65a-d9cc916
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-ServersNegativeTestJSON-1030576039
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-SecurityGroupsNegativeTestJSON-1580739083
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show tempest-ServersTestJSON-732424071
| gigabytes_ceph-ssd    | -1
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
openstack quota show admin
| gigabytes_ceph-ssd    | 1000
                                                                                         |
| snapshots_ceph-ssd    | -1
                                                                                         |
| volumes_ceph-ssd      | -1
                                                                                         |
GregBlow commented 2 weeks ago 
[ceph: root@sv-hdd-12-0 /]# ceph df
--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
hdd    3.6 PiB  1.7 PiB  1.9 PiB   1.9 PiB      52.69
mds     20 TiB   19 TiB  1.0 TiB   1.0 TiB       5.09
ssd    151 TiB   34 TiB  117 TiB   117 TiB      77.58
TOTAL  3.8 PiB  1.8 PiB  2.0 PiB   2.0 PiB      53.41
GregBlow commented 2 weeks ago

qserv looks to have an actual usage of ~2TB at present.

GregBlow commented 2 weeks ago

-1 is not 0

GregBlow commented 2 weeks ago 
(openstack_cli) ubuntu@gb-magnum-2:~$ openstack quota set --gigabytes 0 --volume-type ceph-ssd TiDES
This command currently defaults to '--force' when modifying network quotas. This behavior will change in a future release. Consider explicitly providing '--force' or '--no-force' options to avoid changes in behavior.
(openstack_cli) ubuntu@gb-magnum-2:~$ openstack quota set --gigabytes 0 --volume-type ceph-ssd Somerville-Gaia-Jade
This command currently defaults to '--force' when modifying network quotas. This behavior will change in a future release. Consider explicitly providing '--force' or '--no-force' options to avoid changes in behavior.
Quota gigabytes_ceph-ssd limit must be equal or greater than existing resources. Current usage is 24 and the requested limit is 0. (HTTP 400) (Request-ID: req-fe299d46-f23b-4f35-8ff1-1e2e21e5886a)
(openstack_cli) ubuntu@gb-magnum-2:~$ openstack quota set --gigabytes 24 --volume-type ceph-ssd Somerville-Gaia-Jade
This command currently defaults to '--force' when modifying network quotas. This behavior will change in a future release. Consider explicitly providing '--force' or '--no-force' options to avoid changes in behavior.
GregBlow commented 2 weeks ago 

(openstack_cli) ubuntu@gb-magnum-2:~$ openstack quota set --gigabytes 0 --volume-type ceph-ssd Adler
This command currently defaults to '--force' when modifying network quotas. This behavior will change in a future release. Consider explicitly providing '--force' or '--no-force' options to avoid changes in behavior.
(openstack_cli) ubuntu@gb-magnum-2:~$ openstack quota set --gigabytes 0 --volume-type ceph-ssd Perfsonar
This command currently defaults to '--force' when modifying network quotas. This behavior will change in a future release. Consider explicitly providing '--force' or '--no-force' options to avoid changes in behavior.
(openstack_cli) ubuntu@gb-magnum-2:~$ openstack quota set --gigabytes 0 --volume-type ceph-ssd "VDFS VVV"
This command currently defaults to '--force' when modifying network quotas. This behavior will change in a future release. Consider explicitly providing '--force' or '--no-force' options to avoid changes in behavior.
(openstack_cli) ubuntu@gb-magnum-2:~$ openstack quota set --gigabytes 100 --volume-type ceph-ssd RSP-test
This command currently defaults to '--force' when modifying network quotas. This behavior will change in a future release. Consider explicitly providing '--force' or '--no-force' options to avoid changes in behavior.