Parsing bigip.conf results in UnicodeDecodeError

[runnaman]

Describe the bug Parsing bigip.conf results in UnicodeDecodeError

To Reproduce

1. Run _ucs-modifier -u /UCS/BIGIP_Backup_5-May-2022_15.1.0.4.ucs -m 10.155.28.240 -p default -o /UCS/BIGIP_Backup_5-May-202215.1.0.4-modified.ucs -d

Expected behavior No errors, cleaned UCS is exported.

Desktop (please complete the following information):

• OS: Ubuntu Desktop
• Version 21

Additional context

docker run -v /home/[user]/Downloads:/UCS -it f5devcentral/f5-journeyslab-ucsmodifier:latest

Unable to find image 'f5devcentral/f5-journeyslab-ucsmodifier:latest' locally latest: Pulling from f5devcentral/f5-journeyslab-ucsmodifier Digest: sha256:282299499b062b7dca02ce05eeb217fbe8ace74e2c7204f84c04f2559f882f01 Status: Downloaded newer image for f5devcentral/f5-journeyslab-ucsmodifier:latest

root@6fe2549b78ce:/modifier# ucs-modifier .dockerignore CHANGELOG.md MANIFEST.in README.md setup.cfg setup.py ucs_modifier/ ucs_modifier.egg-info/

root@6fe2549b78ce:/modifier# ucs-modifier -u /UCS/BIGIP_Backup_5-May-2022_15.1.0.4.ucs -m 10.155.28.240 -p default -o /UCS/BIGIP_Backup_5-May-2022_15.1.0.4-modified.ucs -d DEBUG: starting thread (client mode): 0x229f2c10 DEBUG: Local version/idstring: SSH-2.0-paramiko_2.8.1 DEBUG: Remote version/idstring: SSH-2.0-OpenSSH_7.4 INFO: Connected (version 2.0, client OpenSSH_7.4) DEBUG: kex algos:['ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'diffie-hellman-group14-sha1', 'diffie-hellman-group-exchange-sha1'] server key:['ssh-rsa', 'rsa-sha2-512', 'rsa-sha2-256', 'ssh-dss', 'ecdsa-sha2-nistp256', 'ssh-ed25519'] client encrypt:['aes128-gcm@openssh.com', 'aes256-gcm@openssh.com', 'aes128-cbc', 'aes256-cbc', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'aes192-cbc'] server encrypt:['aes128-gcm@openssh.com', 'aes256-gcm@openssh.com', 'aes128-cbc', 'aes256-cbc', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'aes192-cbc'] client mac:['hmac-sha1'] server mac:['hmac-sha1'] client compress:['none', 'zlib@openssh.com'] server compress:['none', 'zlib@openssh.com'] client lang:[''] server lang:[''] kex follows?False DEBUG: Kex agreed: ecdh-sha2-nistp256 DEBUG: HostKey agreed: ssh-ed25519 DEBUG: Cipher agreed: aes128-ctr DEBUG: MAC agreed: hmac-sha1 DEBUG: Compression agreed: none DEBUG: kex engine KexNistp256 specified hash_algo DEBUG: Switch to new keys ... DEBUG: Adding ssh-ed25519 host key for 10.155.28.240: b'7bdf9d5ba3d50ff38739f1d9d447fd89' INFO: Key swap needed with 10.155.28.240, attempting DEBUG: No SSH keys to load DEBUG: Generating a RSA SSH key DEBUG: Host key found for 10.155.28.240, removing from known_hosts DEBUG: starting thread (client mode): 0x22a11e80 DEBUG: Local version/idstring: SSH-2.0-paramiko_2.8.1 DEBUG: Remote version/idstring: SSH-2.0-OpenSSH_7.4 INFO: Connected (version 2.0, client OpenSSH_7.4) DEBUG: kex algos:['ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'diffie-hellman-group14-sha1', 'diffie-hellman-group-exchange-sha1'] server key:['ssh-rsa', 'rsa-sha2-512', 'rsa-sha2-256', 'ssh-dss', 'ecdsa-sha2-nistp256', 'ssh-ed25519'] client encrypt:['aes128-gcm@openssh.com', 'aes256-gcm@openssh.com', 'aes128-cbc', 'aes256-cbc', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'aes192-cbc'] server encrypt:['aes128-gcm@openssh.com', 'aes256-gcm@openssh.com', 'aes128-cbc', 'aes256-cbc', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'aes192-cbc'] client mac:['hmac-sha1'] server mac:['hmac-sha1'] client compress:['none', 'zlib@openssh.com'] server compress:['none', 'zlib@openssh.com'] client lang:[''] server lang:[''] kex follows?False DEBUG: Kex agreed: ecdh-sha2-nistp256 DEBUG: HostKey agreed: ssh-ed25519 DEBUG: Cipher agreed: aes128-ctr DEBUG: MAC agreed: hmac-sha1 DEBUG: Compression agreed: none DEBUG: kex engine KexNistp256 specified hash_algo DEBUG: Switch to new keys ... DEBUG: EOF in transport thread DEBUG: userauth is OK DEBUG: Authentication type (password) not permitted. DEBUG: Allowed methods: ['publickey', 'keyboard-interactive', 'hostbased'] DEBUG: userauth is OK INFO: Authentication (keyboard-interactive) successful! DEBUG: [chan 0] Max packet in: 32768 bytes DEBUG: Received global request hostkeys-00@openssh.com DEBUG: Rejecting hostkeys-00@openssh.com global request from server. DEBUG: [chan 0] Max packet out: 32768 bytes DEBUG: Secsh channel 0 opened. DEBUG: [chan 0] Sesch channel 0 request ok INFO: [chan 0] Opened sftp connection (server version 3) DEBUG: [chan 0] open(b'.ssh/authorized_keys', 'a') DEBUG: [chan 0] open(b'.ssh/authorized_keys', 'a') -> 00000000 DEBUG: [chan 0] close(00000000) DEBUG: starting thread (client mode): 0x22816cd0 DEBUG: Local version/idstring: SSH-2.0-paramiko_2.8.1 DEBUG: Remote version/idstring: SSH-2.0-OpenSSH_7.4 INFO: Connected (version 2.0, client OpenSSH_7.4) DEBUG: kex algos:['ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'diffie-hellman-group14-sha1', 'diffie-hellman-group-exchange-sha1'] server key:['ssh-rsa', 'rsa-sha2-512', 'rsa-sha2-256', 'ssh-dss', 'ecdsa-sha2-nistp256', 'ssh-ed25519'] client encrypt:['aes128-gcm@openssh.com', 'aes256-gcm@openssh.com', 'aes128-cbc', 'aes256-cbc', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'aes192-cbc'] server encrypt:['aes128-gcm@openssh.com', 'aes256-gcm@openssh.com', 'aes128-cbc', 'aes256-cbc', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'aes192-cbc'] client mac:['hmac-sha1'] server mac:['hmac-sha1'] client compress:['none', 'zlib@openssh.com'] server compress:['none', 'zlib@openssh.com'] client lang:[''] server lang:[''] kex follows?False DEBUG: Kex agreed: ecdh-sha2-nistp256 DEBUG: HostKey agreed: ssh-ed25519 DEBUG: Cipher agreed: aes128-ctr DEBUG: MAC agreed: hmac-sha1 DEBUG: Compression agreed: none DEBUG: EOF in transport thread DEBUG: kex engine KexNistp256 specified hash_algo DEBUG: Switch to new keys ... DEBUG: Adding ssh-ed25519 host key for 10.155.28.240: b'7bdf9d5ba3d50ff38739f1d9d447fd89' DEBUG: Trying discovered key b'7be13709afea7b9c7704ab6861e5e759' in /root/.ssh/id_rsa DEBUG: userauth is OK INFO: Authentication (publickey) successful! INFO: Key swap completed successfully DEBUG: RUN: sed -i -r 's/encrypted-password .[{|}].$/encrypted-password $1$salt$IEd.dPRrJY41NYnnaBENQ1/g' /tmp/tmpmf8_yv7f/config/bigip_user.conf OUT: ERR: RC=0 DEBUG: EOF in transport thread Traceback (most recent call last): File "/usr/local/bin/ucs-modifier", line 33, in sys.exit(load_entry_point('ucs-modifier', 'console_scripts', 'ucs-modifier')()) File "/modifier/ucs_modifier/calypso_ucs_modifier.py", line 61, in main modifier = UcsModifier(address=address, extracted_ucs_dir=ucs_dir) File "/modifier/ucs_modifier/modifier/modifier.py", line 39, in init self.config = self.ucs_reader.get_config() File "/modifier/ucs_modifier/journeys/utils/ucs_reader.py", line 32, in get_config return Config.from_dir(self.ucs_path("config")) File "/usr/local/lib/python3.9/site-packages/bigip_parser/config.py", line 193, in from_dir return cls(config, bigdb, dirname) File "/usr/local/lib/python3.9/site-packages/bigip_parser/config.py", line 220, in init raise ConfigLoadError( bigip_parser.errors.ConfigLoadError: Errors found during parsed config load: {'file': 'bigip.conf', 'error': 'UnicodeDecodeError(\'utf-8\', b\'app/eSewa-Org-AWAF\n predicates {\n {\n arguments false\n metric http.cache_control_header_exists\n operator eq\n }\n {\n arguments true\n metric http.referer_header_exists\n operator eq\n }\n {\n arguments 17\n metric http.headers_count\n operator eq\n }\n {\n arguments 2\n metric http.cache_control\n operator not-hashes-to\n }\n {\n arguments /authentication/views/login.html\n metric http.uri_file\n operator hashes-like\n }\n {\n arguments \\'\xfb\xb0W+\n metric http.referer\n operator hashes-like\n }\n {\n arguments 32-47\n metric http.uri_len\n operator between\n }\n {\n arguments accept:accept-encoding:connection:host:user-agent\n metric http.hdrorder\n operator hashes-like\n }\n }\n state mitigate\n type dynamic\n}\nsecurity dos dos-signature /Common/HTTPSig145078799280346564672433429941 {\n alias /Common/HTTPSig145078799280346564672433429941\n creation-time 2021-06-23:11:25:43\n family http\n hardware-offload disabled\n last-modified-time 2021-06-23:11:25:43\n manual-detection-threshold 0\n manual-mitigation-threshold 0\n origin dynamic-bdos\n parent-context /Common/eSewa-Prod-Org-443\n parent-context-type virtual-server\n parent-profile /Common/eSewa-Org-AWAF.app/eSewa-Org-AWAF\n predicates {\n {\n arguments false\n metric http.cache_control_header_exists\n operator eq\n }\n {\n arguments true\n metric http.referer_header_exists\n operator eq\n }\n {\n arguments host:user-agent:accept:accept-encoding:connection\n metric http.hdrorder\n operator hashes-like\n }\n {\n arguments false\n metric http.content_type_header_exists\n operator eq\n }\n {\n arguments 62\n metric http.uri_file\n operator not-hashes-to\n }\n {\n arguments 20\n metric http.uri_file\n operator not-hashes-to\n }\n {\n arguments true\n metric http.cookie_header_exists\n operator eq\n }\n {\n arguments 8\n metric http.uri_file\n operator not-hashes-to\n }\n {\n arguments no-query\n metric http.uri_parameters\n operator eq\n }\n {\n arguments 19\n metric http.uri_file\n operator not-hashes-to\n }\n {\n arguments 40\n metric http.uri_file\n operator not-hashes-to\n }\n {\n arguments true\n metric http.accept_header_exists\n operator eq\n }\n {\n arguments https://org.esewa.com.np/\\n metric http.referer\n operator hashes-like\n }\n {\n arguments 9\n metric http.headers_count\n operator eq\n }\n {\n arguments 32-47\n metric http.uri_len\n operator between\n }\n }\n state mitigate\n type dynamic\n}\nsecurity dos dos-signature /Common/HTTPSig113947594508637337182509868607 {\n alias /Common/HTTPSig113947594508637337182509868607\n creation-time 2022-01-12:12:39:52\n family http\n hardware-offload disabled\n last-modified-time 2022-01-12:12:39:52\n manual-detection-threshold 0\n manual-mitigation-threshold 0\n origin dynamic-bdos\n parent-context /Common/Fonepay-Prod-443\n parent-context-type virtual-server\n parent-profile /Common/Fonepay-AWAF.app/Fonepay-AWAF\n predicates {\n {\n arguments false\n metric http.pragma_header_exists\n operator eq\n }\n {\n arguments false\n metric http.cache_control_header_exists\n operator eq\n }\n {\n arguments false\n metric http.cookie_header_exists\n operator eq\n }\n {\n arguments false\n metric http.authorization_header_exists\n operator eq\n }\n {\n arguments true\n metric http.x_forwarded_for_header_exists\n operator eq\n }\n {\n arguments false\n metric http.content_type_header_exists\n operator eq\n }\n {\n arguments true\n metric http.accept_encoding_header_exists\n operator eq\n }\n {\n arguments true\n metric http.user_agent_header_exists\n operator eq\n }\n {\n arguments true\n metric http.connection_header_exists\n operator eq\n }\n {\n arguments true\n metric http.unknown_header_exists\n operator eq\n }\n {\n arguments 25\n metric http.hdrorder\n operator not-hashes-to\n }\n {\n arguments application\n metric http.accept\n operator doesn\\'t contain\n }\n {\n arguments no-query\n metric http.uri_parameters\n operator neq\n }\n {\n arguments OPTIONS\n metric http.request.method\n operator eq\n }\n {\n arguments true\n metric http.host_header_exists\n operator eq\n }\n {\n arguments false\n metric http.expect_header_exists\n operator eq\n }\n {\n arguments 20\n metric http.hdrorder\n operator not-hashes-to\n }\n {\n arguments false\n metric http.content_length_header_exists\n operator eq\n }\n {\n arguments 36\n metric http.uri_file\n operator not-hashes-to\n }\n {\n arguments https://acquirer.fonepay.com/\\n metric http.referer\n operator hashes-like\n }\n {\n arguments true\n metric http.referer_header_exists\n operator eq\n }\n {\n arguments 14\n metric http.headers_count\n operator eq\n }\n {\n arguments application\n metric http.content_type\n operator doesn\\'t contain"\n }\n {\n arguments true\n metric http.accept_header_exists\n operator eq\n }\n {\n arguments 32-47\n metric http.uri_len\n operator not-between\n }\n {\n arguments 16-31\n metric http.uri_len\n operator not-between\n }\n }\n state mitigate\n type dynamic\n}\nsecurity dos dos-signature /Common/HTTPSig181093786382433429941 {\n alias /Common/HTTPSig181093786382433429941\n creation-time 2021-06-23:12:47:38\n family http\n hardware-offload disabled\n last-modified-time 2021-06-23:12:47:38\n manual-detection-threshold 0\n manual-mitigation-threshold 0\n origin dynamic-bdos\n parent-context /Common/eSewa-Prod-Org-443\n parent-context-type virtual-server\n parent-profile /Common/eSewa-Org-AWAF.app/eSewa-Org-AWAF\n predicates {\n {\n arguments true\n metric http.referer_header_exists\n operator eq\n }\n {\n arguments /authentication/views/login.html\n metric http.uri_file\n operator hashes-like\n }\n {\n arguments https://org.esewa.com.np/\\n metric http.referer\n operator hashes-like\n }\n {\n arguments 32-47\n metric http.uri_len\n operator between\n }\n {\n arguments 12\n metric http.headers_count\n operator eq\n }\n {\n arguments host:connecti\', 676, 677, \'invalid start byte\')', 'line': None}

[mnaumiuk]

Filled internal: SOLUTIONS-6876

[mnaumiuk]

fixed in the newest version 1.0.4