search

Intel-bigdata / SSM

Smart Storage Management for Big Data, a comprehensive hot/cold data optimized solution
Apache License 2.0
133 stars 67 forks source link

Fix the namespace fetch issue in the whitelist #2201

Closed lipppppp closed 3 years ago

lipppppp commented 3 years ago

The inotify event will be filtered by whitelist when whitelist is enabled. If user create some files which is not in the whitelist, these inotify events won't be recorded in SSM. But if user wants to add these new files path to whitelist after creating, their inotify events already filtered which leads to their metadata missing in the file table.

lipppppp commented 3 years ago

In order to solve this problem, I made the following design. If you have any good suggestions, welcome to discuss. 1.Create a new file 'smart.properties' to store smart.last.fetch.dirs property. This property records last fetch path. 2.Check whether the path in the current whitelist is the same as the previous one at each restart. If the whitelist changed, namespace will be fetched fully again. 3.Finally, update smart.last.fetch.dirs property to current whitelist path.

PHILO-HE commented 3 years ago

If fetched dirs are maintained in local master smart server, what if master is down and standby smart server takes the role? Maybe, we have to maintain fetched dirs in database which is accessible by two or more smart servers.

lipppppp commented 3 years ago

Good suggestions. If HA happens, new active smart server will fetch namespace fully and create a new smart.properties in the local dir. This causes unnecessary workload. Maintaining fetched dirs in database is a better choice.

PHILO-HE commented 3 years ago

The patch looks good to me. Will commit soon.