search

fastogt / fastonosql

FastoNoSQL is a crossplatform Redis, Memcached, SSDB, LevelDB, RocksDB, UnQLite, LMDB, ForestDB, Pika, Dynomite, KeyDB GUI management tool.
https://fastonosql.com
GNU General Public License v3.0
1.17k stars 96 forks source link

Fails to expand keys appropriately. #47

Open rquadling opened 6 years ago

rquadling commented 6 years ago

Hi.

My log says ...

12:03:19.155: INFO function error: A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE
12:03:33.535: INFO function error: A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE

My command history is ...

[MEMCACHED] 12:03:19.155: STATS
[MEMCACHED] 12:03:33.534: STATS
[MEMCACHED] 12:03:45.094: SCAN 0 MATCH * COUNT 1000
[MEMCACHED] 12:03:45.106: TTL [DT:categoryController:tickets:670:9522][1]
[MEMCACHED] 12:03:45.108: TTL [DT:ticketsController:cats:264][1]
[MEMCACHED] 12:03:45.112: TTL [DT:ticketsController:cats:670][1]
[MEMCACHED] 12:04:20.984: SCAN 0 MATCH * COUNT 1000
[MEMCACHED] 12:04:20.997: TTL [DT:categoryController:tickets:670:9522][1]
[MEMCACHED] 12:04:21.001: TTL [DT:ticketsController:cats:264][1]
[MEMCACHED] 12:04:21.006: TTL [DT:ticketsController:cats:670][1]
[MEMCACHED] 12:10:16.805: SCAN 0 MATCH * COUNT 1000
[MEMCACHED] 12:10:16.828: TTL [DT:categoryController:tickets:670:9522][1]
[MEMCACHED] 12:10:16.828: TTL [DT:ticketsController:cats:264][1]
[MEMCACHED] 12:10:16.828: TTL [DT:ticketsController:cats:670][1]

I'm using DoctrineCache, so the format of [xxxx][n] is their's. I only supply the xxxx part.

The UI shows ...

image

What's missing is the expansion of the remainder of the keys.

It would be nice to be able to extend the key splitting with known key generators.

DoctrineCache's structure is %s[%s][%d], where the 3 parts are namespace (optional), key (user supplied and if multi part can have any sort of separation, and version (a number).

I'm guessing other systems generate keys in a different way. Maybe having a UI in Fastonosql that allows multiple splitters as well as ignore string sequences.

So, : and ][ are splitters, but [ and ] are ignored. Hmm. Obviously doing the split then ignore works for this pattern. May not work for all though.

Happy to help in any way I can on this.

topilski commented 6 years ago

Hi @rquadling, now FastoNoSQL have only (combobox)namespace separator when user create connection, can you describe in more detail how use can input complex namespace separator?

topilski commented 6 years ago

Your keys should display like in redis, why i see folder i don't know i need to try reproduce your image. untitled

topilski commented 6 years ago

Also if TTL expired, key will removed from tree, but folders can exists.

rquadling commented 6 years ago

The cache hasn't expired in my testing. That's part of what I'm using Fastonosql for. To see what is happening in the cache. The error that is generated in the logs tells me something is wrong. I do not know what it is wrong. Our application is operating well with the cache. Our code (PHP based and using DoctrineCache) says we are getting our data from the cache.

Our code only supplies part of the key. The remainder is provided/wrapped by the caching code we use.

topilski commented 6 years ago

I will try to reproduce your case, thank you!

rquadling commented 6 years ago

If we had the capability of defining multiple separators, then I could add : and ][.

This would then cause the tree to look like

[DT
  categoryController
    tickets
      670
        9532
          0]

And then removing [ and ] would fix the first and last elements, though they could remain if that causes issues.

topilski commented 6 years ago

12:03:19.155: INFO function error: A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE 12:03:33.535: INFO function error: A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE

I fixed this warnings, it was my mistake. About separators, how user can specify this separators in settings, in your case you have 2 pair of separators(: and ][) but if someone will want 10 separators, how i can handle user input?

rquadling commented 6 years ago

I would have a string list that they can add entries to, as well as a character removal/clean option.

So. Maybe a UI like ...

Add separator: [          ]     [Add]
Separators:
+----------------------------------------+
| :                             [Remove] |
| ][                            [Remove] |
|                                        |
|                                        |
|                                        |
|                                        |
|                                        |
|                                        |
|                                        |
|                                        |
+----------------------------------------+

Remove from key(s): [          ]

And document that splitting takes place before removal.

So, for me the Remove from key(s): would look like.

Remove from key(s): [[]        ]

This would mean that the DoctrineCache key of [DT:categoryController:tickets:670:9522][1] is treated as a set of DT, categoryController, tickets, 670, 9522, and 1.

topilski commented 6 years ago

Thank you for description, i will think about your solution. In current moment you can select separator which not containing in your keys, and working like with common keys.

rquadling commented 6 years ago

If you find that there are a LOT of standardised structures, then maybe having a checkbox of supported splits/cleans.

So, maybe ...

Keys generated by
(*) DoctrineCache

Add separator: [          ]     [Add]
Separators:
+----------------------------------------+
| :                             [Remove] |
| ][                            [Remove] |
|                                        |
|                                        |
|                                        |
|                                        |
|                                        |
|                                        |
|                                        |
|                                        |
+----------------------------------------+

Remove from key(s): [          ]
topilski commented 6 years ago

Fixed, and new release uploaded, in namespace combobox you can write separators which you want, in your case it will be :[] , all this characters will be delimiters.

topilski commented 6 years ago

ping

rquadling commented 6 years ago

I'll get back to you next week - on a different development for a couple of days.

rquadling commented 6 years ago

Hi.

Keys expanding correctly now.

I'm using :[]| as my separators and getting ...

image

But when I try to get the value for any of those keys ...

Logs:

12:36:07.093: GET function error: NOT FOUND
12:36:16.832: GET function error: NOT FOUND

Commands:

[MEMCACHED] 12:36:07.093: GET [DT:categoryController|tickets|659|4110][1]
[MEMCACHED] 12:36:16.832: GET [DT:ticketsController|cats|659][1]

So. Progress!! But still not usable.

topilski commented 6 years ago

Hi, thank you for feedback, will continue working on it.

topilski commented 6 years ago

Also can you please check if key [DT:categoryController|tickets|659|4110][1] exist in your db?

rquadling commented 6 years ago

I've just loaded about 5 more pages which will cache their data.

I am getting very peculiar responses visually in Fastonosql.

I'm now only getting folders. No keys.

Also, a key with a 4 hour TTL is not being presented correctly at all.

[MEMCACHED] 14:41:07.839: SCAN 0 MATCH * COUNT 1000
[MEMCACHED] 14:41:07.847: TTL [DT:all-time-total][1]
[MEMCACHED] 14:41:07.848: TTL [DT:categoryController|tickets|659|4051][1]
[MEMCACHED] 14:41:07.851: TTL [DT:categoryController|tickets|659|4037][1]
[MEMCACHED] 14:41:07.861: TTL [DT:categoryController|tickets|659|4053][1]
[MEMCACHED] 14:41:07.862: TTL [DT:categoryController|tickets|659|4050][1]
[MEMCACHED] 14:41:07.862: TTL [DT:categoryController|tickets|659|4045][1]
[MEMCACHED] 14:41:07.865: TTL [DT:categoryController|tickets|659|4110][1]
[MEMCACHED] 14:41:07.869: TTL [DT:ticketsController|cats|659][1]

The all-time-total is going to be around for several hours. Doesn't display as a key. Only a folder.

Using :[]| as separators.

I wonder if the split is resulting in zero length strings when it gets to ][ in the key?

image

rquadling commented 6 years ago

What is interesting is that SOMETIMES when I load database, I get a key and then it all reverts to folders and no keys.

rquadling commented 6 years ago

I've taken a look at the data using ...

memdump --servers=localhost | xargs -L1 -I% sh -c 'echo "get %" | nc localhost 11211'

and got ...

VALUE [DT:all-time-total][1] 0 12
147627475.52
END
VALUE [DT:categoryController|tickets|659|4051][1] 52 1654
?x??Ymo?6???8?C??#?ʼnHS?s?:A??/?t??H?FRv???}GRv????0
                                                   ?` p$??????Qb?0??駱(D??n?X??>??$lD??J?mct?:a?_?9?????.????v?V[?D??`??=y??
???V???u??                                                                                                                  :#??u??e|a$?9
??5?2?+.? ܰ莭?P-?xV?&pʚ-??????????%?:J??Z??ƭ̘???$?REI?
                                 ??bԑ??Rx?? l?????B?
                                                    ?9F???{/???b
                                                                ?.#?$HE??j̋-???ɫ&p?Ele?@Vh,2?p?7؆???!L?
                                                                                                      .
                                                                                                       ?\?aֳ???Yf]?B??[???)B$???m?7?a"?.?fU??
                                                                                                                                            0[Y(??E(?
                                                                                                                                                     T#?ekU?I?R!??(?????E???m?9?"??w?D  S,??? 9?x??d?????6? ?-2Ւi
?:?K????Q~??鳧?Nj%??q?|%??]Ҫ?&"g??J?
[snipped]

So this is telling me that there is data.

I know the first one will be around for about 4 hours, so is an easy enough test.

The others have a ttl of about a minute, but I can easily modify that.

But memdump is showing me data (though the larger data is compressed).

topilski commented 6 years ago

This happens because keys have time to live. in FastoNoSQL if key was removed but have folders, folders will not cleaned.