search

graphite-project / carbon

Carbon is one of the components of Graphite, and is responsible for receiving metrics over the network and writing them down to disk using a storage backend.
http://graphite.readthedocs.org/
Apache License 2.0
1.5k stars 490 forks source link

[BUG] Manhole fails to find public cert #948

Closed itinneed2022 closed 10 months ago

itinneed2022 commented 11 months ago

Please report bug only for carbon component here, if you want to to do so for Graphite, please use graphite-web repo

Describe the bug When trying to activate the manhole config, the certificate produced cannot be found at the location given.

To Reproduce Produce ssh host key with: ckeygen -t rsa -f /app/graphite/conf/ssh_host_key_rsa Enable Manhole in carbon.conf:

ENABLE_MANHOLE = True
MANHOLE_INTERFACE = 127.0.0.1
MANHOLE_PORT = 7222
MANHOLE_USER = admin
MANHOLE_PUBLIC_KEY = ssh-rsa AAAAB3NzaC1yc2EAAAABiwAaAIEAoxN0sv/e4eZCPpi3N3KYvyzRaBaMeS2RsOQ/cDuKv11dlNzVeiyc3RFmCv5Rjwn/lQ79y0zyHxw67qLyhQ/kDzINc4cY41ivuQXm2tPmgvexdrBv5nsfEpjs3gLZfJnyvlcVyWK/lId8WUvEWSWHTzsbtmXAF2raJMdgLTbQ8wE=
MANHOLE_HOST_KEY_DIR = /app/graphite/conf/

Restart carbon-cache systemctl restart carbon-cache

Expected behavior An ssh port should be exposed that I can tunnel into the carbon-cache daemons and check their metrics.

Environment (please complete the following information):

Additional context This is the error from the carbon-cache startup log:

16/08/2023 16:57:01 :: Traceback (most recent call last):
16/08/2023 16:57:01 ::   File "bin/carbon-cache.py", line 32, in <module>
16/08/2023 16:57:01 ::     run_twistd_plugin(__file__)
16/08/2023 16:57:01 ::   File "/app/graphite/lib/carbon/util.py", line 140, in run_twistd_plugin
16/08/2023 16:57:01 ::     runApp(config)
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/scripts/twistd.py", line 29, in runApp
16/08/2023 16:57:01 ::     runner.run()
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/application/app.py", line 374, in run
16/08/2023 16:57:01 ::     self.postApplication()
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/scripts/_twistd_unix.py", line 254, in postApplication
16/08/2023 16:57:01 ::     self.startApplication(self.application)
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/scripts/_twistd_unix.py", line 444, in startApplication
16/08/2023 16:57:01 ::     service.IService(application).privilegedStartService()
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/application/service.py", line 271, in privilegedStartService
16/08/2023 16:57:01 ::     service.privilegedStartService()
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/application/service.py", line 271, in privilegedStartService
16/08/2023 16:57:01 ::     service.privilegedStartService()
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/application/internet.py", line 115, in privilegedStartService
16/08/2023 16:57:01 ::     self._port = self._getPort()
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/application/internet.py", line 144, in _getPort
16/08/2023 16:57:01 ::     )(*self.args, **self.kwargs)
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/internet/posixbase.py", line 565, in listenTCP
16/08/2023 16:57:01 ::     p.startListening()
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/internet/tcp.py", line 1356, in startListening
16/08/2023 16:57:01 ::     self.factory.doStart()
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/internet/protocol.py", line 73, in doStart
16/08/2023 16:57:01 ::     self.startFactory()
16/08/2023 16:57:01 ::   File "/usr/local/lib/python3.6/site-packages/twisted/conch/ssh/factory.py", line 44, in startFactory
16/08/2023 16:57:01 ::     raise error.ConchError("no host public keys, failing")
16/08/2023 16:57:01 :: twisted.conch.error.ConchError: ('no host public keys, failing', None)
deniszh commented 11 months ago

Thanks for reporting. That's need to be checked and addressed, we'll do that soon.

ср, 23 авг. 2023 г., 17:11 itinneed2022 @.***>:

Please report bug only for carbon component here, if you want to to do so for Graphite, please use graphite-web repo https://github.com/graphite-project/graphite-web/issues/new/choose

Describe the bug When trying to activate the manhole config, the certificate produced cannot be found at the location given.

To Reproduce Produce ssh host key with: ckeygen -t rsa -f /app/graphite/conf/ssh_host_key_rsa Enable Manhole in carbon.conf:

ENABLE_MANHOLE = True MANHOLE_INTERFACE = 127.0.0.1 MANHOLE_PORT = 7222 MANHOLE_USER = admin MANHOLE_PUBLIC_KEY = ssh-rsa AAAAB3NzaC1yc2EAAAABiwAaAIEAoxN0sv/e4eZCPpi3N3KYvyzRaBaMeS2RsOQ/cDuKv11dlNzVeiyc3RFmCv5Rjwn/lQ79y0zyHxw67qLyhQ/kDzINc4cY41ivuQXm2tPmgvexdrBv5nsfEpjs3gLZfJnyvlcVyWK/lId8WUvEWSWHTzsbtmXAF2raJMdgLTbQ8wE= MANHOLE_HOST_KEY_DIR = /app/graphite/conf/

Restart carbon-cache systemctl restart carbon-cache

Expected behavior An ssh port should be exposed that I can tunnel into the carbon-cache daemons and check their metrics.

Environment (please complete the following information):

  • OS flavor: Red Hat 7
  • Graphite version 1.1.0
  • Setup type: From Source
  • Python version: 3.6

Additional context This is the error from the carbon-cache startup log:

16/08/2023 16:57:01 :: Traceback (most recent call last): 16/08/2023 16:57:01 :: File "bin/carbon-cache.py", line 32, in 16/08/2023 16:57:01 :: run_twistd_plugin(file) 16/08/2023 16:57:01 :: File "/app/graphite/lib/carbon/util.py", line 140, in run_twistd_plugin 16/08/2023 16:57:01 :: runApp(config) 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/scripts/twistd.py", line 29, in runApp 16/08/2023 16:57:01 :: runner.run() 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/application/app.py", line 374, in run 16/08/2023 16:57:01 :: self.postApplication() 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/scripts/_twistd_unix.py", line 254, in postApplication 16/08/2023 16:57:01 :: self.startApplication(self.application) 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/scripts/_twistd_unix.py", line 444, in startApplication 16/08/2023 16:57:01 :: service.IService(application).privilegedStartService() 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/application/service.py", line 271, in privilegedStartService 16/08/2023 16:57:01 :: service.privilegedStartService() 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/application/service.py", line 271, in privilegedStartService 16/08/2023 16:57:01 :: service.privilegedStartService() 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/application/internet.py", line 115, in privilegedStartService 16/08/2023 16:57:01 :: self._port = self._getPort() 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/application/internet.py", line 144, in _getPort 16/08/2023 16:57:01 :: )(*self.args, **self.kwargs) 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/internet/posixbase.py", line 565, in listenTCP 16/08/2023 16:57:01 :: p.startListening() 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/internet/tcp.py", line 1356, in startListening 16/08/2023 16:57:01 :: self.factory.doStart() 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/internet/protocol.py", line 73, in doStart 16/08/2023 16:57:01 :: self.startFactory() 16/08/2023 16:57:01 :: File "/usr/local/lib/python3.6/site-packages/twisted/conch/ssh/factory.py", line 44, in startFactory 16/08/2023 16:57:01 :: raise error.ConchError("no host public keys, failing") 16/08/2023 16:57:01 :: twisted.conch.error.ConchError: ('no host public keys, failing', None)

— Reply to this email directly, view it on GitHub https://github.com/graphite-project/carbon/issues/948, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJLTVXSLUSYNAT5ZUUTINDXWYMSFANCNFSM6AAAAAA33UJSPA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

deniszh commented 10 months ago

@itinneed2022 : that's happening because of Twisted 16.1 and later is no longer uses a hardcoded SSH host key pair. So, for Twisted 16.1 and newer you need to set MANHOLE_PUBLIC_KEY = None and it works. Probably easy fix in the code.

deniszh commented 10 months ago

Fixed for master in https://github.com/graphite-project/carbon/pull/950. But MANHOLE_PUBLIC_KEY = None also works.

itinneed2022 commented 10 months ago

@deniszh I must be missing something. I'm still getting the same error:

28/08/2023 09:15:39 :: twistd 22.4.0 (/usr/bin/python3.6 3.6.8) starting up.
28/08/2023 09:15:39 :: reactor class: twisted.internet.epollreactor.EPollReactor.
28/08/2023 09:15:39 :: ServerFactory starting on 7002
28/08/2023 09:15:39 :: Starting factory <twisted.internet.protocol.ServerFactory object at 0x7f94c3ae1fd0>
28/08/2023 09:15:39 :: ConchFactory starting on 7222
28/08/2023 09:15:39 :: Starting factory <twisted.conch.manhole_ssh.ConchFactory object at 0x7f94c1c30198>
28/08/2023 09:15:39 :: Traceback (most recent call last):
28/08/2023 09:15:39 ::   File "bin/carbon-cache.py", line 32, in <module>
28/08/2023 09:15:39 ::     run_twistd_plugin(__file__)
28/08/2023 09:15:39 ::   File "/app/graphite/lib/carbon/util.py", line 140, in run_twistd_plugin
28/08/2023 09:15:39 ::     runApp(config)
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/scripts/twistd.py", line 29, in runApp
28/08/2023 09:15:39 ::     runner.run()
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/application/app.py", line 374, in run
28/08/2023 09:15:39 ::     self.postApplication()
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/scripts/_twistd_unix.py", line 254, in postApplication
28/08/2023 09:15:39 ::     self.startApplication(self.application)
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/scripts/_twistd_unix.py", line 444, in startApplication
28/08/2023 09:15:39 ::     service.IService(application).privilegedStartService()
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/application/service.py", line 271, in privilegedStartService
28/08/2023 09:15:39 ::     service.privilegedStartService()
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/application/service.py", line 271, in privilegedStartService
28/08/2023 09:15:39 ::     service.privilegedStartService()
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/application/internet.py", line 115, in privilegedStartService
28/08/2023 09:15:39 ::     self._port = self._getPort()
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/application/internet.py", line 144, in _getPort
28/08/2023 09:15:39 ::     )(*self.args, **self.kwargs)
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/internet/posixbase.py", line 565, in listenTCP
28/08/2023 09:15:39 ::     p.startListening()
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/internet/tcp.py", line 1356, in startListening
28/08/2023 09:15:39 ::     self.factory.doStart()
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/internet/protocol.py", line 73, in doStart
28/08/2023 09:15:39 ::     self.startFactory()
28/08/2023 09:15:39 ::   File "/usr/local/lib/python3.6/site-packages/twisted/conch/ssh/factory.py", line 44, in startFactory
28/08/2023 09:15:39 ::     raise error.ConchError("no host keys, failing")
28/08/2023 09:15:39 :: twisted.conch.error.ConchError: ('no host keys, failing', None)

Here is the Manhole config in carbon.conf:

# The manhole interface allows you to SSH into the carbon daemon
# and get a python interpreter. BE CAREFUL WITH THIS! If you do
# something like time.sleep() in the interpreter, the whole process
# will sleep! This is *extremely* helpful in debugging, assuming
# you are familiar with the code. If you are not, please don't
# mess with this, you are asking for trouble :)
#
# You need the bcrypt, cryptography and pyasn1 python modules installed for
# manhole to work.
#
# Generate host keys with:
# `ckeygen -t rsa -f /example/host_keys/ssh_host_key_rsa`
#
ENABLE_MANHOLE = True
MANHOLE_INTERFACE = 127.0.0.1
MANHOLE_PORT = 7222
MANHOLE_USER = admin
MANHOLE_PUBLIC_KEY = None
MANHOLE_HOST_KEY_DIR=/app/graphite/example/host_keys

If I did not specify the manhole_host_key_dir carbon threw a different error:

Aug 28 09:21:30 myservercarbon-cache[45220]: /usr/local/lib/python3.6/site-packages/twisted/conch/ssh/transport.py:101: CryptographyDeprecationWarning: CAST5 has been deprecated
Aug 28 09:21:30 myservercarbon-cache[45220]: b"cast128-cbc": (algorithms.CAST5, 16, modes.CBC),
Aug 28 09:21:30 myservercarbon-cache[45220]: /usr/local/lib/python3.6/site-packages/twisted/conch/ssh/transport.py:106: CryptographyDeprecationWarning: Blowfish has been deprecated
Aug 28 09:21:30 myservercarbon-cache[45220]: b"blowfish-ctr": (algorithms.Blowfish, 16, modes.CTR),
Aug 28 09:21:30 myservercarbon-cache[45220]: /usr/local/lib/python3.6/site-packages/twisted/conch/ssh/transport.py:107: CryptographyDeprecationWarning: CAST5 has been deprecated
Aug 28 09:21:30 myservercarbon-cache[45220]: b"cast128-ctr": (algorithms.CAST5, 16, modes.CTR),
Aug 28 09:21:30 myservercarbon-cache[45220]: Starting carbon-cache (instance e)
Aug 28 09:21:30 myservercarbon-cache[45220]: MANHOLE_HOST_KEY_DIR not defined
Aug 28 09:21:30 myservercarbon-cache[45220]: [FAILED]
deniszh commented 10 months ago

What's in /app/graphite/example/host_keys directory? You should have ssh_host_key_rsa and ssh_host_key_rsa.pub there, with proper permissions, available for read for carbon user.

itinneed2022 commented 10 months ago 
-rwxrwxrwx. 1 root root  886 Aug 16 15:12 ssh_host_key_rsa
-rwxrwxrwx. 1 root root  234 Aug 16 15:12 ssh_host_key_rsa.pub