search

open-io / oio-sds

High Performance Software-Defined Object Storage for Big Data and AI, that supports Amazon S3 and Openstack Swift
https://www.openio.io
Other
668 stars 92 forks source link

Difference in objects count #1237

Open couloum opened 7 years ago

couloum commented 7 years ago

Hi,

On my instance, I saw a difference in objects count between the following commands:

Exemple:

$ openio container list
+-----------------------------------------------------------------------------------------------------------------------------+---------+-------+
| Name                                                                                                                        |   Bytes | Count |
+-----------------------------------------------------------------------------------------------------------------------------+---------+-------+
| XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX                                                      |  299053 |    11 |
+-----------------------------------------------------------------------------------------------------------------------------+---------+-------+
$ openio container show XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+----------------+------------------------------------------------------------------------+
| Field          | Value                                                                  |
+----------------+------------------------------------------------------------------------+
| account        | XXXXXXXXXX                                                             |
| base_name      | 2C717B78826F7AA7E4B2C14A7646D0F84F397006811A0F7ABAD4D3421E13CF13.1     |
| bytes_usage    | 327.337KB                                                              |
| container      | XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX |
| ctime          | 1489059129                                                             |
| max_versions   | Namespace default                                                      |
| objects        | 13                                                                     |
| quota          | Namespace default                                                      |
| storage_policy | Namespace default                                                      |
+----------------+------------------------------------------------------------------------+

As you can see the number of objects is different in both commands. I've run the command several time to make sure it was not because some objects has been added in between.

Here are my packages version:

$ apt list --installed | grep openio
diamond-openio/now 20161019.gitc789102-1 all [installed,local]
openio-asn1c/xenial,now 0.9.27.1-1 amd64 [installed,automatic]
openio-gridinit/xenial,now 1.6-2 amd64 [installed]
openio-sds/xenial,now 4.1.4-1 amd64 [installed]
openio-sds-dbg/xenial,now 4.1.4-1 amd64 [installed]
openio-sds-logrotate/xenial,now 1.3-1 all [installed]
openio-sds-replicator/now 20170228.git250b866-1 all [installed,local]
openio-sds-rsyslog/xenial,now 1.2-2 all [installed]

Thank you

fvennetier commented 7 years ago

Hello. You can try openio account refresh XXXXXXXXXX. This will ask all containers to send their current state to the account service (where the list of containers is stored).

fvennetier commented 7 years ago

If you have only one inconsistent container, you can simply run openio container touch XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX (assuming XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX is the name of the container).

couloum commented 7 years ago

Hi,

Thanks for your answer.

I did not tried the openio account refresh command as I have around 2 millions containers in production and was not sure of the performance impact of such command on the system. Do you know if I can safely run it without production impact?

Regarding the openio container touch command, this does not seem to work:

$ openio container touch XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
vcfr1qx01p>~# openio container list --marker XXXXXXXXX --limit 1
+----------------------------------------------------------------------------+-------+-------+
| Name                                                                       | Bytes | Count |
+----------------------------------------------------------------------------+-------+-------+
| XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX            |  4834 |     1 |
+----------------------------------------------------------------------------+-------+-------+
vcfr1qx01p>~#openio container show XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+----------------+--------------------------------------------------------------------+
| Field          | Value                                                              |
+----------------+--------------------------------------------------------------------+
| account        | XXXXXXXXXX                                                         |
| base_name      | 7BC38B7E7C7C1281E5807F27F783B206350C27F5E7412AC34FC253C75F32B742.1 |
| bytes_usage    | 0B                                                                 |
| container      | XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX    |
| ctime          | 1505127022                                                         |
| max_versions   | Namespace default                                                  |
| objects        | 0                                                                  |
| quota          | Namespace default                                                  |
| storage_policy | Namespace default                                                  |
+----------------+--------------------------------------------------------------------+
fvennetier commented 7 years ago

The account update relies on the event system. openio container touch will ask the meta2 service to send an event to a beanstalkd queue. oio-event-agent will eventually read that event from the queue and update the account service.

If there are too many events on the platform, this may take some time.

If for some reason the account service is down when the event is treated, the event will be "buried". You can run openio events stats to see the number of buried events, and you can run openio events exhume to replay these events (1000 events per call).

And yes, openio account refresh may have a production impact, by increasing the load and memory usage.