kytos-ng / storehouse

Persistence NApp with support for multiple backends
https://kytos-ng.github.io/api/storehouse.html
MIT License
0 stars 2 forks source link

Corrupted file blocks storehouse from loading persisted data #9

Open italovalcy opened 2 years ago

italovalcy commented 2 years ago

Hi,

I'm not sure the conditions that led to the following error, but after stopping Kytos and starting it again, the following exception was raised:

2022-02-03 06:37:23,003 - INFO [kytos.core.controller] (MainThread) Loading NApp kytos/storehouse
2022-02-03 06:37:23,136 - INFO [kytos.napps.kytos/storehouse] (MainThread) Loading 'filesystem' backend...
2022-02-03 06:37:23,157 - CRITICAL [kytos.core.controller] (MainThread) NApp initialization failed: kytos/storehouse
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/kytos/core/controller.py", line 840, in load_napp
    napp = napp_module.Main(controller=self)
  File "/usr/local/lib/python3.9/dist-packages/kytos/core/napps/base.py", line 193, in __init__
    self.setup()
  File "/var/lib/kytos/napps/kytos/storehouse/main.py", line 93, in setup
    self.create_cache()
  File "/var/lib/kytos/napps/kytos/storehouse/main.py", line 107, in create_cache
    box = self.backend.retrieve(namespace, box_id)
  File "/var/lib/kytos/napps/../napps/kytos/storehouse/backends/fs.py", line 115, in retrieve
    return self._load_from_file(destination)
  File "/var/lib/kytos/napps/../napps/kytos/storehouse/backends/fs.py", line 83, in _load_from_file
    data = pickle.load(load_file)
EOFError: Ran out of input

As a consequence, many stored data was not available after that error, including the one being loaded and the ones that should be loaded right after.

Trying to load some of the stored boxes manually showed the following output:

$ cat /home/italo/load-from-storehouse.py
#!/usr/bin/python3
import pickle
import sys
import json

sys.path.append("/var/lib/kytos")
sys.path.append("/src/kytos")
sys.path.append("/src/python-openflow")

def _load_from_file(filename):
    with open(filename, 'rb') as load_file:
        data = pickle.load(load_file)
    return data

if len(sys.argv) < 2:
    print("USAGE: %s <STOREHOUSEFILE>" % sys.argv[0])
    sys.exit(1)

filename = sys.argv.pop()
data = _load_from_file(filename).data
print(json.dumps(data))
$ for file in $(find . -type f); do echo $file; /home/italo/load-from-storehouse.py $file > /dev/null; done
./kytos/storehouse/kytos.topology.switches.metadata/62ab57a5ec004d43a96ec30ea77fab05
./kytos/storehouse/kytos.topology.interfaces.metadata/ee3d9c4e0cd944e4b7cfd9903d0fd112
./kytos/storehouse/kytos.mef_eline.circuits/bd99034467c24d2da632cab41a1292f7
./kytos/storehouse/kytos.flow.persistence/6c7ec1d0977c4b378ac9672afe6d2c0f
Traceback (most recent call last):
  File "/home/italo/load-from-storehouse.py", line 20, in <module>
    data = _load_from_file(filename).data
  File "/home/italo/load-from-storehouse.py", line 12, in _load_from_file
    data = pickle.load(load_file)
EOFError: Ran out of input
./kytos/storehouse/kytos.topology.status/43d46c47ddbd4e82a3d4d13f72a1215f
./kytos/storehouse/kytos.topology.links.metadata/759f7e48a90e4e0dab9de2de93c74a28

The scenario in which this error happened was: 1) running Kytos-ng with docker image amlight/kytos-prod:2022.2rc1 2) Kytos is running with the console enable (kytosd -f) inside a tmux session (default behavior for the docker init script) 3) Kytos is killed with the following command:

docker exec kytos1 bash -c "pkill kytosd && sleep 5 && pkill -9 kytosd"
sleep 30
docker exec kytos1 bash -c "tmux new-session -d -s kytosserver kytosd -f"

This issue seems to be quite common in the above sequence of steps. Out of 10 executions, all of them resulted in the same error:

$ for RUN in 20220125155444 20220125162453 20220125173256 20220125184145 20220125194856 20220125205809 20220125220828 20220125231803 20220126002704 20220126014311; do echo "-- $RUN"; rm -rf tmp-results/*; tar -xzf /home/kytos/storehouse-$RUN.tgz -C tmp-results/; for file in $(find tmp-results/home/kytos/storehouse/ -type f); do echo $file; /home/italo/load-from-storehouse.py $file > /dev/null; done; done
-- 20220125155444
tmp-results/home/kytos/storehouse/kytos.topology.switches.metadata/721b96ee29ab40a2ac3dfabc0f319783
tmp-results/home/kytos/storehouse/kytos.topology.interfaces.metadata/5c961e4b8f6944509d81cc0bc7a2c8a5
tmp-results/home/kytos/storehouse/kytos.mef_eline.circuits/4974b838c9024264aaffea1f08147679
tmp-results/home/kytos/storehouse/kytos.flow.persistence/e4d505c6f2724ec1978427de935a8a54
Traceback (most recent call last):
  File "/home/italo/load-from-storehouse.py", line 20, in <module>
    data = _load_from_file(filename).data
  File "/home/italo/load-from-storehouse.py", line 12, in _load_from_file
    data = pickle.load(load_file)
EOFError: Ran out of input
tmp-results/home/kytos/storehouse/kytos.topology.status/6a7f2f919c014f6d9080ed40b07d9a1f
tmp-results/home/kytos/storehouse/kytos.topology.links.metadata/77a9d5c81f434240b1623b581db7f221
-- 20220125162453
tmp-results/home/kytos/storehouse/kytos.topology.switches.metadata/f1a04fcd193747a3812811fb5c914575
tmp-results/home/kytos/storehouse/kytos.topology.interfaces.metadata/c904d0c7a33746ed9c72d43800d95fc4
tmp-results/home/kytos/storehouse/kytos.mef_eline.circuits/01d810b012c340649e7fc676127d775e
tmp-results/home/kytos/storehouse/kytos.flow.persistence/543888efbf3b4885a9d9e6f814f5a935
Traceback (most recent call last):
  File "/home/italo/load-from-storehouse.py", line 20, in <module>
    data = _load_from_file(filename).data
  File "/home/italo/load-from-storehouse.py", line 12, in _load_from_file
    data = pickle.load(load_file)
EOFError: Ran out of input
tmp-results/home/kytos/storehouse/kytos.topology.status/0beb347e83e74dde8b5ea922312da457
tmp-results/home/kytos/storehouse/kytos.topology.links.metadata/ca8c1fa37d804a57b99d3b04d33ab2f5
-- 20220125173256
tmp-results/home/kytos/storehouse/kytos.topology.switches.metadata/494f843a420a401ea1eccfda98ca70a1
tmp-results/home/kytos/storehouse/kytos.topology.interfaces.metadata/13de57366e574689af6542e3589a1d06
tmp-results/home/kytos/storehouse/kytos.mef_eline.circuits/95fbf649e3cf48f7940fb603202c007a
tmp-results/home/kytos/storehouse/kytos.flow.persistence/b40124b46ac64883b3844043dd78d6fa
Traceback (most recent call last):
  File "/home/italo/load-from-storehouse.py", line 20, in <module>
    data = _load_from_file(filename).data
  File "/home/italo/load-from-storehouse.py", line 12, in _load_from_file
    data = pickle.load(load_file)
EOFError: Ran out of input
tmp-results/home/kytos/storehouse/kytos.topology.status/f514ac36d8014260879958c4e6e4f8b7
tmp-results/home/kytos/storehouse/kytos.topology.links.metadata/b668c5a0b2674903be3affddfa6f90b1
-- 20220125184145
tmp-results/home/kytos/storehouse/kytos.topology.switches.metadata/a850ad10c8d04fcc9d9ce5a16f97e8c2
tmp-results/home/kytos/storehouse/kytos.topology.interfaces.metadata/858a8c59ea1a4faa8fd52839e200a07e
tmp-results/home/kytos/storehouse/kytos.mef_eline.circuits/b2248c6971aa42de927c05f505742907
tmp-results/home/kytos/storehouse/kytos.flow.persistence/77553ee4b2fa451fa7c8526d02df6332
Traceback (most recent call last):
  File "/home/italo/load-from-storehouse.py", line 20, in <module>
    data = _load_from_file(filename).data
  File "/home/italo/load-from-storehouse.py", line 12, in _load_from_file
    data = pickle.load(load_file)
EOFError: Ran out of input
tmp-results/home/kytos/storehouse/kytos.topology.status/d5b2bef427274f8bb03a1f35bcce789e
tmp-results/home/kytos/storehouse/kytos.topology.links.metadata/51be6015c5f846728fa519361a1f9303
-- 20220125194856
tmp-results/home/kytos/storehouse/kytos.topology.switches.metadata/7bc92de633bd4cb5bee42c10f0cbfa67
tmp-results/home/kytos/storehouse/kytos.topology.interfaces.metadata/b7224f6bb333469189fa603d6a19c148
tmp-results/home/kytos/storehouse/kytos.mef_eline.circuits/5e94410954d64b5494fef9f776edfc7a
tmp-results/home/kytos/storehouse/kytos.flow.persistence/5faa8b08b7054a61afdb78028c12e632
Traceback (most recent call last):
  File "/home/italo/load-from-storehouse.py", line 20, in <module>
    data = _load_from_file(filename).data
  File "/home/italo/load-from-storehouse.py", line 12, in _load_from_file
    data = pickle.load(load_file)
EOFError: Ran out of input
tmp-results/home/kytos/storehouse/kytos.topology.status/684011bf81284d08a0c80b6f9d64ae80
tmp-results/home/kytos/storehouse/kytos.topology.links.metadata/9e72f274a25f4061b7a3eb63016c0d03
-- 20220125205809
tmp-results/home/kytos/storehouse/kytos.topology.switches.metadata/bf94b82dc0de4a7e9b96e9a07ae53665
tmp-results/home/kytos/storehouse/kytos.topology.interfaces.metadata/22f9dbd3d59f4e5d9d389312089fbb8e
tmp-results/home/kytos/storehouse/kytos.mef_eline.circuits/d45ad29c544d4f79960bdccb50235bd3
tmp-results/home/kytos/storehouse/kytos.flow.persistence/6782c83e8f58430d8ff6108919be88e0
Traceback (most recent call last):
  File "/home/italo/load-from-storehouse.py", line 20, in <module>
    data = _load_from_file(filename).data
  File "/home/italo/load-from-storehouse.py", line 12, in _load_from_file
    data = pickle.load(load_file)
EOFError: Ran out of input
tmp-results/home/kytos/storehouse/kytos.topology.status/f79e51f8cd304d029f564e0a343d6284
tmp-results/home/kytos/storehouse/kytos.topology.links.metadata/d5808c8392a24e9ab3b45172bb6e103f
-- 20220125220828
tmp-results/home/kytos/storehouse/kytos.topology.switches.metadata/8cea3b78f8b84f42b51830faed13f3a0
tmp-results/home/kytos/storehouse/kytos.topology.interfaces.metadata/93866475f5774979b6da69e68e40e97b
tmp-results/home/kytos/storehouse/kytos.mef_eline.circuits/d019d6a8259f41a0a4d1c32bc8639f16
tmp-results/home/kytos/storehouse/kytos.flow.persistence/7a38d268a3a84a358f86bbbc11255e09
Traceback (most recent call last):
  File "/home/italo/load-from-storehouse.py", line 20, in <module>
    data = _load_from_file(filename).data
  File "/home/italo/load-from-storehouse.py", line 12, in _load_from_file
    data = pickle.load(load_file)
EOFError: Ran out of input
tmp-results/home/kytos/storehouse/kytos.topology.status/c7e5bb2d54b4424dbb5bcb6a29de6b70
tmp-results/home/kytos/storehouse/kytos.topology.links.metadata/34e65e1af7594de08ce1a585cd0faedf
-- 20220125231803
tmp-results/home/kytos/storehouse/kytos.topology.switches.metadata/56b090806b6a4669894f5f48ad3ecf00
tmp-results/home/kytos/storehouse/kytos.topology.interfaces.metadata/9d7fff2866de4ab9b8c87b1682df0b28
tmp-results/home/kytos/storehouse/kytos.mef_eline.circuits/7dfa8309e59645468d5aae0c841bf64d
tmp-results/home/kytos/storehouse/kytos.flow.persistence/3c2b9df475134cf28b04ff194970144b
Traceback (most recent call last):
  File "/home/italo/load-from-storehouse.py", line 20, in <module>
    data = _load_from_file(filename).data
  File "/home/italo/load-from-storehouse.py", line 12, in _load_from_file
    data = pickle.load(load_file)
EOFError: Ran out of input
tmp-results/home/kytos/storehouse/kytos.topology.status/649e5ec028f14ec3a44328312d4c7134
tmp-results/home/kytos/storehouse/kytos.topology.links.metadata/8bc763896421495091be9f4c15290bdd
-- 20220126002704
tmp-results/home/kytos/storehouse/kytos.topology.switches.metadata/0a228d590b614ad0aa8e51f36d538aba
tmp-results/home/kytos/storehouse/kytos.topology.interfaces.metadata/04769a280a6642928412dabcf5d7da84
tmp-results/home/kytos/storehouse/kytos.mef_eline.circuits/09c0cd35c141453aabfc3a54fc419572
tmp-results/home/kytos/storehouse/kytos.flow.persistence/2752d520d353439baf36ca10bdb246ed
Traceback (most recent call last):
  File "/home/italo/load-from-storehouse.py", line 20, in <module>
    data = _load_from_file(filename).data
  File "/home/italo/load-from-storehouse.py", line 12, in _load_from_file
    data = pickle.load(load_file)
EOFError: Ran out of input
tmp-results/home/kytos/storehouse/kytos.topology.status/3040374189b94ce4904dfd675f152cac
tmp-results/home/kytos/storehouse/kytos.topology.links.metadata/9144d12745b84526993fa28809de3f21
-- 20220126014311
tmp-results/home/kytos/storehouse/kytos.topology.switches.metadata/85133f5ebf44405f9c796db1266e1171
tmp-results/home/kytos/storehouse/kytos.topology.interfaces.metadata/e4c3c1c68a374796a7a81a7d222ad457
tmp-results/home/kytos/storehouse/kytos.mef_eline.circuits/53f963da99f447ab8d5dea80427e2dd4
tmp-results/home/kytos/storehouse/kytos.flow.persistence/9ab3d10c94604635ab98d9657fb5f751
Traceback (most recent call last):
  File "/home/italo/load-from-storehouse.py", line 20, in <module>
    data = _load_from_file(filename).data
  File "/home/italo/load-from-storehouse.py", line 12, in _load_from_file
    data = pickle.load(load_file)
EOFError: Ran out of input
tmp-results/home/kytos/storehouse/kytos.topology.status/36d98857eb9a4035b7f7be000a6db63f
tmp-results/home/kytos/storehouse/kytos.topology.links.metadata/ffb2b5d442014f8990cdbf467aca83c2
italovalcy commented 2 years ago

Another test showed that this is not specific to the environment I'm running at. Steps to reproduce:

  1. Run kytos from docker image amlight/kytos:latest (Ex: docker run -d --name k99 --privileged amlight/kytos:latest)
  2. Start mininet with a basic topology (Ex: docker exec -it k99 tmux new-sess -d "mn --topo=linear,10 --controller=remote,ip=127.0.0.1"
  3. Enable switches, links and create 100 EVCs:
    docker exec -it k99 bash
    IP=127.0.0.1; for sw in $(curl -s http://$IP:8181/api/kytos/topology/v3/switches | jq -r '.switches[].id'); do curl -H 'Content-type: application/json' -X POST http://$IP:8181/api/kytos/topology/v3/switches/$sw/enable; curl -H 'Content-type: application/json' -X POST http://$IP:8181/api/kytos/topology/v3/interfaces/switch/$sw/enable; done
    IP=127.0.0.1; for l in $(curl -s http://$IP:8181/api/kytos/topology/v3/links | jq -r '.links[].id'); do curl -H 'Content-type: application/json' -X POST http://$IP:8181/api/kytos/topology/v3/links/$l/enable; done
    for vlan in $(seq 401 500); do curl -H 'Content-type: application/json' -X POST http://127.0.0.1:8181/api/kytos/mef_eline/v2/evc -d '{"name": "evc-vlan-'$vlan'", "dynamic_backup_path": true, "enabled": true, "uni_z": {"tag": {"value": '$vlan', "tag_type": 1}, "interface_id": "00:00:00:00:00:00:00:01:1"}, "uni_a": {"tag": {"value": '$vlan', "tag_type": 1}, "interface_id": "00:00:00:00:00:00:00:0a:1"}}'; done
  4. Using the same load-from-storehouse.py script above, try to load the saved data from storehouse:
    # ./load-from-storehouse.py /var/tmp/kytos/storehouse/kytos.flow.persistence/*
    Traceback (most recent call last):
    File "//./load-from-storehouse.py", line 20, in <module>
    data = _load_from_file(filename).data
    File "//./load-from-storehouse.py", line 12, in _load_from_file
    data = pickle.load(load_file)
    EOFError: Ran out of input
italovalcy commented 2 years ago

Continuing with the debugging of this issue, I managed to save the flow persistence box using Kytos console and the results are very different:

tmux a -t kytosserver
kytos $> import pickle
kytos $> f = open("/tmp/bla", "wb")
kytos $> pickle.dump(controller.napps[ ('kytos', 'flow_manager')].storehouse.box, f)
kytos $> f.close()

Then, from the command line:

# ls -l /tmp/bla
-rw-r--r-- 1 root root 433062 Feb  4 14:48 /tmp/bla
# ls -l /var/tmp/kytos/storehouse/kytos.flow.persistence/2baf479b304348bea24ae830a3801659
-rw-r--r-- 1 root root 196638 Feb  4 14:24 /var/tmp/kytos/storehouse/kytos.flow.persistence/2baf479b304348bea24ae830a3801659
# /load-from-storehouse.py /tmp/bla > /dev/null
# /load-from-storehouse.py /var/tmp/kytos/storehouse/kytos.flow.persistence/2baf479b304348bea24ae830a3801659 > /dev/null
Traceback (most recent call last):
  File "/load-from-storehouse.py", line 20, in <module>
    data = _load_from_file(filename).data
  File "/load-from-storehouse.py", line 12, in _load_from_file
    data = pickle.load(load_file)
EOFError: Ran out of input

Looking into the size of the file, it seems to be a problem in the way flow_manager is persisting data. I'll open another issue over kytos-ng/flow_manager.

Anyway, kytos-ng/storehouse should have a way to deal with failures like the one we just observed. One approach could be to backup the current file and create an empty one? These along with an exception handler when loading a file, of course.

viniarck commented 2 years ago

For future readers, I'm setting this as wontfix since this NApp has been deprecated see PR #15