search

graphite-project / carbon

Carbon is one of the components of Graphite, and is responsible for receiving metrics over the network and writing them down to disk using a storage backend.
http://graphite.readthedocs.org/
Apache License 2.0
1.51k stars 490 forks source link

[BUG] Manhole fails to find public cert #948

Closed itinneed2022 closed 1 year ago

itinneed2022 commented 1 year ago

Please report bug only for carbon component here, if you want to to do so for Graphite, please use graphite-web repo

Describe the bug When trying to activate the manhole config, the certificate produced cannot be found at the location given.

To Reproduce Produce ssh host key with: ckeygen -t rsa -f /app/graphite/conf/ssh_host_key_rsa Enable Manhole in carbon.conf:

ENABLE_MANHOLE = True
MANHOLE_INTERFACE = 127.0.0.1
MANHOLE_PORT = 7222
MANHOLE_USER = admin
MANHOLE_PUBLIC_KEY = ssh-rsa AAAAB3NzaC1yc2EAAAABiwAaAIEAoxN0sv/e4eZCPpi3N3KYvyzRaBaMeS2RsOQ/cDuKv11dlNzVeiyc3RFmCv5Rjwn/lQ79y0zyHxw67qLyhQ/kDzINc4cY41ivuQXm2tPmgvexdrBv5nsfEpjs3gLZfJnyvlcVyWK/lId8WUvEWSWHTzsbtmXAF2raJMdgLTbQ8wE=
MANHOLE_HOST_KEY_DIR = /app/graphite/conf/

Restart carbon-cache systemctl restart carbon-cache

Expected behavior An ssh port should be exposed that I can tunnel into the carbon-cache daemons and check their metrics.

Environment (please complete the following information):

Additional context This is the error from the carbon-cache startup log:

16/08/2023 16:57:01 :: Traceback (most recent call last):
16/08/2023 16:57:01 ::   File "bin/carbon-cache.py", line 32, in <module>
16/08/2023 16:57:01 ::     run_twistd_plugin(__file__)
16/08/2023 16:57:01 ::   File "/app/graphite/lib/carbon/util.py", line 140, in run_twistd_plugin
16/08/2023 16:57:01 ::     runApp(config)
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/scripts/twistd.py", line 29, in runApp
16/08/2023 16:57:01 ::     runner.run()
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/application/app.py", line 374, in run
16/08/2023 16:57:01 ::     self.postApplication()
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/scripts/_twistd_unix.py", line 254, in postApplication
16/08/2023 16:57:01 ::     self.startApplication(self.application)
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/scripts/_twistd_unix.py", line 444, in startApplication
16/08/2023 16:57:01 ::     service.IService(application).privilegedStartService()
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/application/service.py", line 271, in privilegedStartService
16/08/2023 16:57:01 ::     service.privilegedStartService()
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/application/service.py", line 271, in privilegedStartService
16/08/2023 16:57:01 ::     service.privilegedStartService()
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/application/internet.py", line 115, in privilegedStartService
16/08/2023 16:57:01 ::     self._port = self._getPort()
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/application/internet.py", line 144, in _getPort
16/08/2023 16:57:01 ::     )(*self.args, **self.kwargs)
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/internet/posixbase.py", line 565, in listenTCP
16/08/2023 16:57:01 ::     p.startListening()
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/internet/tcp.py", line 1356, in startListening
16/08/2023 16:57:01 ::     self.factory.doStart()
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/internet/protocol.py", line 73, in doStart
16/08/2023 16:57:01 ::     self.startFactory()
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/conch/ssh/factory.py", line 44, in startFactory
16/08/2023 16:57:01 ::     raise error.ConchError("no host public keys, failing")
16/08/2023 16:57:01 :: twisted.conch.error.ConchError: ('no host public keys, failing', None)
deniszh commented 1 year ago

Thanks for reporting. That's need to be checked and addressed, we'll do that soon.

ср, 23 авг. 2023 г., 17:11 itinneed2022 @.***>:

Please report bug only for carbon component here, if you want to to do so for Graphite, please use graphite-web repo https://github.com/graphite-project/graphite-web/issues/new/choose

Describe the bug When trying to activate the manhole config, the certificate produced cannot be found at the location given.

To Reproduce Produce ssh host key with: ckeygen -t rsa -f /app/graphite/conf/ssh_host_key_rsa Enable Manhole in carbon.conf:

ENABLE_MANHOLE = True MANHOLE_INTERFACE = 127.0.0.1 MANHOLE_PORT = 7222 MANHOLE_USER = admin MANHOLE_PUBLIC_KEY = ssh-rsa AAAAB3NzaC1yc2EAAAABiwAaAIEAoxN0sv/e4eZCPpi3N3KYvyzRaBaMeS2RsOQ/cDuKv11dlNzVeiyc3RFmCv5Rjwn/lQ79y0zyHxw67qLyhQ/kDzINc4cY41ivuQXm2tPmgvexdrBv5nsfEpjs3gLZfJnyvlcVyWK/lId8WUvEWSWHTzsbtmXAF2raJMdgLTbQ8wE= MANHOLE_HOST_KEY_DIR = /app/graphite/conf/

Restart carbon-cache systemctl restart carbon-cache

Expected behavior An ssh port should be exposed that I can tunnel into the carbon-cache daemons and check their metrics.

Environment (please complete the following information):

  • OS flavor: Red Hat 7
  • Graphite version 1.1.0
  • Setup type: From Source
  • Python version: 3.6

Additional context This is the error from the carbon-cache startup log:

16/08/2023 16:57:01 :: Traceback (most recent call last): 16/08/2023 16:57:01 :: File "bin/carbon-cache.py", line 32, in 16/08/2023 16:57:01 :: run_twistd_plugin(file) 16/08/2023 16:57:01 :: File "/app/graphite/lib/carbon/util.py", line 140, in run_twistd_plugin 16/08/2023 16:57:01 :: runApp(config) 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/scripts/twistd.py", line 29, in runApp 16/08/2023 16:57:01 :: runner.run() 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/application/app.py", line 374, in run 16/08/2023 16:57:01 :: self.postApplication() 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/scripts/_twistd_unix.py", line 254, in postApplication 16/08/2023 16:57:01 :: self.startApplication(self.application) 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/scripts/_twistd_unix.py", line 444, in startApplication 16/08/2023 16:57:01 :: service.IService(application).privilegedStartService() 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/application/service.py", line 271, in privilegedStartService 16/08/2023 16:57:01 :: service.privilegedStartService() 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/application/service.py", line 271, in privilegedStartService 16/08/2023 16:57:01 :: service.privilegedStartService() 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/application/internet.py", line 115, in privilegedStartService 16/08/2023 16:57:01 :: self._port = self._getPort() 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/application/internet.py", line 144, in _getPort 16/08/2023 16:57:01 :: )(*self.args, **self.kwargs) 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/internet/posixbase.py", line 565, in listenTCP 16/08/2023 16:57:01 :: p.startListening() 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/internet/tcp.py", line 1356, in startListening 16/08/2023 16:57:01 :: self.factory.doStart() 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/internet/protocol.py", line 73, in doStart 16/08/2023 16:57:01 :: self.startFactory() 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/conch/ssh/factory.py", line 44, in startFactory 16/08/2023 16:57:01 :: raise error.ConchError("no host public keys, failing") 16/08/2023 16:57:01 :: twisted.conch.error.ConchError: ('no host public keys, failing', None)

— Reply to this email directly, view it on GitHub https://github.com/graphite-project/carbon/issues/948, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJLTVXSLUSYNAT5ZUUTINDXWYMSFANCNFSM6AAAAAA33UJSPA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

deniszh commented 1 year ago

@itinneed2022 : that's happening because of Twisted 16.1 and later is no longer uses a hardcoded SSH host key pair. So, for Twisted 16.1 and newer you need to set MANHOLE_PUBLIC_KEY = None and it works. Probably easy fix in the code.

deniszh commented 1 year ago

Fixed for master in https://github.com/graphite-project/carbon/pull/950. But MANHOLE_PUBLIC_KEY = None also works.

itinneed2022 commented 1 year ago

@deniszh I must be missing something. I'm still getting the same error:

28/08/2023 09:15:39 :: twistd 22.4.0 (/usr/bin/python3.6 3.6.8) starting up.
28/08/2023 09:15:39 :: reactor class: twisted.internet.epollreactor.EPollReactor.
28/08/2023 09:15:39 :: ServerFactory starting on 7002
28/08/2023 09:15:39 :: Starting factory <twisted.internet.protocol.ServerFactory object at 0x7f94c3ae1fd0>
28/08/2023 09:15:39 :: ConchFactory starting on 7222
28/08/2023 09:15:39 :: Starting factory <twisted.conch.manhole_ssh.ConchFactory object at 0x7f94c1c30198>
28/08/2023 09:15:39 :: Traceback (most recent call last):
28/08/2023 09:15:39 ::   File "bin/carbon-cache.py", line 32, in <module>
28/08/2023 09:15:39 ::     run_twistd_plugin(__file__)
28/08/2023 09:15:39 ::   File "/app/graphite/lib/carbon/util.py", line 140, in run_twistd_plugin
28/08/2023 09:15:39 ::     runApp(config)
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/scripts/twistd.py", line 29, in runApp
28/08/2023 09:15:39 ::     runner.run()
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/application/app.py", line 374, in run
28/08/2023 09:15:39 ::     self.postApplication()
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/scripts/_twistd_unix.py", line 254, in postApplication
28/08/2023 09:15:39 ::     self.startApplication(self.application)
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/scripts/_twistd_unix.py", line 444, in startApplication
28/08/2023 09:15:39 ::     service.IService(application).privilegedStartService()
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/application/service.py", line 271, in privilegedStartService
28/08/2023 09:15:39 ::     service.privilegedStartService()
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/application/service.py", line 271, in privilegedStartService
28/08/2023 09:15:39 ::     service.privilegedStartService()
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/application/internet.py", line 115, in privilegedStartService
28/08/2023 09:15:39 ::     self._port = self._getPort()
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/application/internet.py", line 144, in _getPort
28/08/2023 09:15:39 ::     )(*self.args, **self.kwargs)
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/internet/posixbase.py", line 565, in listenTCP
28/08/2023 09:15:39 ::     p.startListening()
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/internet/tcp.py", line 1356, in startListening
28/08/2023 09:15:39 ::     self.factory.doStart()
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/internet/protocol.py", line 73, in doStart
28/08/2023 09:15:39 ::     self.startFactory()
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/conch/ssh/factory.py", line 44, in startFactory
28/08/2023 09:15:39 ::     raise error.ConchError("no host keys, failing")
28/08/2023 09:15:39 :: twisted.conch.error.ConchError: ('no host keys, failing', None)

Here is the Manhole config in carbon.conf:

# The manhole interface allows you to SSH into the carbon daemon
# and get a python interpreter. BE CAREFUL WITH THIS! If you do
# something like time.sleep() in the interpreter, the whole process
# will sleep! This is *extremely* helpful in debugging, assuming
# you are familiar with the code. If you are not, please don't
# mess with this, you are asking for trouble :)
#
# You need the bcrypt, cryptography and pyasn1 python modules installed for
# manhole to work.
#
# Generate host keys with:
# `ckeygen -t rsa -f /example/host_keys/ssh_host_key_rsa`
#
ENABLE_MANHOLE = True
MANHOLE_INTERFACE = 127.0.0.1
MANHOLE_PORT = 7222
MANHOLE_USER = admin
MANHOLE_PUBLIC_KEY = None
MANHOLE_HOST_KEY_DIR=/app/graphite/example/host_keys

If I did not specify the manhole_host_key_dir carbon threw a different error:

Aug 28 09:21:30 myservercarbon-cache[45220]: /usr/local/lib/python3.6/site-packages/twisted/conch/ssh/transport.py:101: CryptographyDeprecationWarning: CAST5 has been deprecated
Aug 28 09:21:30 myservercarbon-cache[45220]: b"cast128-cbc": (algorithms.CAST5, 16, modes.CBC),
Aug 28 09:21:30 myservercarbon-cache[45220]: /usr/local/lib/python3.6/site-packages/twisted/conch/ssh/transport.py:106: CryptographyDeprecationWarning: Blowfish has been deprecated
Aug 28 09:21:30 myservercarbon-cache[45220]: b"blowfish-ctr": (algorithms.Blowfish, 16, modes.CTR),
Aug 28 09:21:30 myservercarbon-cache[45220]: /usr/local/lib/python3.6/site-packages/twisted/conch/ssh/transport.py:107: CryptographyDeprecationWarning: CAST5 has been deprecated
Aug 28 09:21:30 myservercarbon-cache[45220]: b"cast128-ctr": (algorithms.CAST5, 16, modes.CTR),
Aug 28 09:21:30 myservercarbon-cache[45220]: Starting carbon-cache (instance e)
Aug 28 09:21:30 myservercarbon-cache[45220]: MANHOLE_HOST_KEY_DIR not defined
Aug 28 09:21:30 myservercarbon-cache[45220]: [FAILED]
deniszh commented 1 year ago

What's in /app/graphite/example/host_keys directory? You should have ssh_host_key_rsa and ssh_host_key_rsa.pub there, with proper permissions, available for read for carbon user.

itinneed2022 commented 1 year ago 
-rwxrwxrwx. 1 root root  886 Aug 16 15:12 ssh_host_key_rsa
-rwxrwxrwx. 1 root root  234 Aug 16 15:12 ssh_host_key_rsa.pub