Closed eddiejaoude closed 3 years ago
I think the Discord bot does not have access to connect to the API on Kubernetes. We might need to not use the external url api.eddiehub.org
but an internal one (possibly with some config).
@sidpalas any recommendations on this?
Log output...
node:internal/errors:642
const ex = new Error(msg);
^
Error: socket hang up
at connResetException (node:internal/errors:642:14)
at Socket.socketOnEnd (node:_http_client:496:23)
at Socket.emit (node:events:381:22)
at endReadableNT (node:internal/streams/readable:1307:12)
at processTicksAndRejections (node:internal/process/task_queues:81:21) {
code: 'ECONNRESET',
config: {
url: 'http://api.eddiehub.org',
method: 'get',
headers: {
Accept: 'application/json, text/plain, */*',
'User-Agent': 'axios/0.21.1'
},
transformRequest: [ [Function: transformRequest] ],
transformResponse: [ [Function: transformResponse] ],
timeout: 0,
adapter: [Function: httpAdapter],
xsrfCookieName: 'XSRF-TOKEN',
xsrfHeaderName: 'X-XSRF-TOKEN',
maxContentLength: -1,
maxBodyLength: -1,
validateStatus: [Function: validateStatus],
data: undefined
},
request: <ref *2> Writable {
_writableState: WritableState {
objectMode: false,
highWaterMark: 16384,
finalCalled: false,
needDrain: false,
ending: false,
ended: false,
finished: false,
destroyed: false,
decodeStrings: true,
defaultEncoding: 'utf8',
length: 0,
writing: false,
corked: 0,
sync: true,
bufferProcessing: false,
onwrite: [Function: bound onwrite],
writecb: null,
writelen: 0,
afterWriteTickInfo: null,
buffered: [],
bufferedIndex: 0,
allBuffers: true,
allNoop: true,
pendingcb: 0,
constructed: true,
prefinished: false,
errorEmitted: false,
emitClose: true,
autoDestroy: true,
errored: null,
closed: false,
closeEmitted: false,
[Symbol(kOnFinished)]: []
},
_events: [Object: null prototype] {
response: [Function: handleResponse],
error: [Function: handleRequestError]
},
_eventsCount: 2,
_maxListeners: undefined,
_options: {
maxRedirects: 21,
maxBodyLength: 10485760,
protocol: 'http:',
path: '/',
method: 'GET',
headers: {
Accept: 'application/json, text/plain, */*',
'User-Agent': 'axios/0.21.1'
},
agent: undefined,
agents: { http: undefined, https: undefined },
auth: undefined,
hostname: 'api.eddiehub.org',
port: null,
nativeProtocols: {
'http:': {
_connectionListener: [Function: connectionListener],
METHODS: [
'ACL', 'BIND', 'CHECKOUT',
'CONNECT', 'COPY', 'DELETE',
'GET', 'HEAD', 'LINK',
'LOCK', 'M-SEARCH', 'MERGE',
'MKACTIVITY', 'MKCALENDAR', 'MKCOL',
'MOVE', 'NOTIFY', 'OPTIONS',
'PATCH', 'POST', 'PRI',
'PROPFIND', 'PROPPATCH', 'PURGE',
'PUT', 'REBIND', 'REPORT',
'SEARCH', 'SOURCE', 'SUBSCRIBE',
'TRACE', 'UNBIND', 'UNLINK',
'UNLOCK', 'UNSUBSCRIBE'
],
STATUS_CODES: {
'100': 'Continue',
'101': 'Switching Protocols',
'102': 'Processing',
'103': 'Early Hints',
'200': 'OK',
'201': 'Created',
'202': 'Accepted',
'203': 'Non-Authoritative Information',
'204': 'No Content',
'205': 'Reset Content',
'206': 'Partial Content',
'207': 'Multi-Status',
'208': 'Already Reported',
'226': 'IM Used',
'300': 'Multiple Choices',
'301': 'Moved Permanently',
'302': 'Found',
'303': 'See Other',
'304': 'Not Modified',
'305': 'Use Proxy',
'307': 'Temporary Redirect',
'308': 'Permanent Redirect',
'400': 'Bad Request',
'401': 'Unauthorized',
'402': 'Payment Required',
'403': 'Forbidden',
'404': 'Not Found',
'405': 'Method Not Allowed',
'406': 'Not Acceptable',
'407': 'Proxy Authentication Required',
'408': 'Request Timeout',
'409': 'Conflict',
'410': 'Gone',
'411': 'Length Required',
'412': 'Precondition Failed',
'413': 'Payload Too Large',
'414': 'URI Too Long',
'415': 'Unsupported Media Type',
'416': 'Range Not Satisfiable',
'417': 'Expectation Failed',
'418': "I'm a Teapot",
'421': 'Misdirected Request',
'422': 'Unprocessable Entity',
'423': 'Locked',
'424': 'Failed Dependency',
'425': 'Too Early',
'426': 'Upgrade Required',
'428': 'Precondition Required',
'429': 'Too Many Requests',
'431': 'Request Header Fields Too Large',
'451': 'Unavailable For Legal Reasons',
'500': 'Internal Server Error',
'501': 'Not Implemented',
'502': 'Bad Gateway',
'503': 'Service Unavailable',
'504': 'Gateway Timeout',
'505': 'HTTP Version Not Supported',
'506': 'Variant Also Negotiates',
'507': 'Insufficient Storage',
'508': 'Loop Detected',
'509': 'Bandwidth Limit Exceeded',
'510': 'Not Extended',
'511': 'Network Authentication Required'
},
Agent: [Function: Agent] { defaultMaxSockets: Infinity },
ClientRequest: [Function: ClientRequest],
IncomingMessage: [Function: IncomingMessage],
OutgoingMessage: [Function: OutgoingMessage],
Server: [Function: Server],
ServerResponse: [Function: ServerResponse],
createServer: [Function: createServer],
validateHeaderName: [Function: __node_internal_],
validateHeaderValue: [Function: __node_internal_],
get: [Function: get],
request: [Function: request],
maxHeaderSize: [Getter],
globalAgent: [Getter/Setter]
},
'https:': {
Agent: [Function: Agent],
globalAgent: Agent {
_events: [Object: null prototype],
_eventsCount: 2,
_maxListeners: undefined,
defaultPort: 443,
protocol: 'https:',
options: [Object],
requests: {},
sockets: {},
freeSockets: {},
keepAliveMsecs: 1000,
keepAlive: false,
maxSockets: Infinity,
maxFreeSockets: 256,
scheduling: 'lifo',
maxTotalSockets: Infinity,
totalSocketCount: 0,
maxCachedSessions: 100,
_sessionCache: [Object],
[Symbol(kCapture)]: false
},
Server: [Function: Server],
createServer: [Function: createServer],
get: [Function: get],
request: [Function: request]
}
},
pathname: '/'
},
_ended: true,
_ending: true,
_redirectCount: 0,
_redirects: [],
_requestBodyLength: 0,
_requestBodyBuffers: [],
_onNativeResponse: [Function (anonymous)],
_currentRequest: <ref *1> ClientRequest {
_events: [Object: null prototype] {
response: [Function: bound onceWrapper] {
listener: [Function (anonymous)]
},
abort: [Function (anonymous)],
aborted: [Function (anonymous)],
connect: [Function (anonymous)],
error: [Function (anonymous)],
socket: [Function (anonymous)],
timeout: [Function (anonymous)]
},
_eventsCount: 7,
_maxListeners: undefined,
outputData: [],
outputSize: 0,
writable: true,
destroyed: false,
_last: true,
chunkedEncoding: false,
shouldKeepAlive: false,
_defaultKeepAlive: true,
useChunkedEncodingByDefault: false,
sendDate: false,
_removedConnection: false,
_removedContLen: false,
_removedTE: false,
_contentLength: 0,
_hasBody: true,
_trailer: '',
finished: true,
_headerSent: true,
_closed: false,
socket: Socket {
connecting: false,
_hadError: true,
_parent: null,
_host: 'api.eddiehub.org',
_readableState: ReadableState {
objectMode: false,
highWaterMark: 16384,
buffer: BufferList { head: null, tail: null, length: 0 },
length: 0,
pipes: [],
flowing: true,
ended: true,
endEmitted: true,
reading: false,
constructed: true,
sync: false,
needReadable: false,
emittedReadable: false,
readableListening: false,
resumeScheduled: false,
errorEmitted: false,
emitClose: false,
autoDestroy: true,
destroyed: true,
errored: null,
closed: true,
closeEmitted: true,
defaultEncoding: 'utf8',
awaitDrainWriters: null,
multiAwaitDrain: false,
readingMore: false,
decoder: null,
encoding: null,
[Symbol(kPaused)]: false
},
_events: [Object: null prototype] {
end: [ [Function: onReadableStreamEnd], [Function: socketOnEnd] ],
free: [Function: onFree],
close: [ [Function: onClose], [Function: socketCloseListener] ],
timeout: [Function: onTimeout],
agentRemove: [Function: onRemove],
error: [Function: socketErrorListener],
data: [Function: socketOnData],
drain: [Function: ondrain]
},
_eventsCount: 8,
_maxListeners: undefined,
_writableState: WritableState {
objectMode: false,
highWaterMark: 16384,
finalCalled: false,
needDrain: false,
ending: false,
ended: false,
finished: false,
destroyed: true,
decodeStrings: false,
defaultEncoding: 'utf8',
length: 0,
writing: false,
corked: 0,
sync: false,
bufferProcessing: false,
onwrite: [Function: bound onwrite],
writecb: null,
writelen: 0,
afterWriteTickInfo: null,
buffered: [],
bufferedIndex: 0,
allBuffers: true,
allNoop: true,
pendingcb: 0,
constructed: true,
prefinished: false,
errorEmitted: false,
emitClose: false,
autoDestroy: true,
errored: null,
closed: true,
closeEmitted: true,
[Symbol(kOnFinished)]: []
},
allowHalfOpen: false,
_sockname: null,
_pendingData: null,
_pendingEncoding: '',
server: null,
_server: null,
parser: null,
_httpMessage: [Circular *1],
write: [Function: writeAfterFIN],
[Symbol(async_id_symbol)]: 13022,
[Symbol(kHandle)]: null,
[Symbol(kSetNoDelay)]: false,
[Symbol(lastWriteQueueSize)]: 0,
[Symbol(timeout)]: null,
[Symbol(kBuffer)]: null,
[Symbol(kBufferCb)]: null,
[Symbol(kBufferGen)]: null,
[Symbol(kCapture)]: false,
[Symbol(kBytesRead)]: 0,
[Symbol(kBytesWritten)]: 130
},
_header: 'GET / HTTP/1.1\r\n' +
'Accept: application/json, text/plain, */*\r\n' +
'User-Agent: axios/0.21.1\r\n' +
'Host: api.eddiehub.org\r\n' +
'Connection: close\r\n' +
'\r\n',
_keepAliveTimeout: 0,
_onPendingData: {},
agent: Agent {
_events: [Object: null prototype] {
free: [Function (anonymous)],
newListener: [Function: maybeEnableKeylog]
},
_eventsCount: 2,
_maxListeners: undefined,
defaultPort: 80,
protocol: 'http:',
options: { path: null },
requests: {},
sockets: { 'api.eddiehub.org:80:': [ [Socket] ] },
freeSockets: {},
keepAliveMsecs: 1000,
keepAlive: false,
maxSockets: Infinity,
maxFreeSockets: 256,
scheduling: 'lifo',
maxTotalSockets: Infinity,
totalSocketCount: 1,
[Symbol(kCapture)]: false
},
socketPath: undefined,
method: 'GET',
maxHeaderSize: undefined,
insecureHTTPParser: undefined,
path: '/',
_ended: false,
res: null,
aborted: false,
timeoutCb: null,
upgradeOrConnect: false,
parser: null,
maxHeadersCount: null,
reusedSocket: false,
host: 'api.eddiehub.org',
protocol: 'http:',
_redirectable: [Circular *2],
[Symbol(kCapture)]: false,
[Symbol(kNeedDrain)]: false,
[Symbol(corked)]: 0,
[Symbol(kOutHeaders)]: [Object: null prototype] {
accept: [ 'Accept', 'application/json, text/plain, */*' ],
'user-agent': [ 'User-Agent', 'axios/0.21.1' ],
host: [ 'Host', 'api.eddiehub.org' ]
}
},
_currentUrl: 'http://api.eddiehub.org/',
[Symbol(kCapture)]: false
},
response: undefined,
isAxiosError: true,
toJSON: [Function: toJSON]
}
npm notice
npm notice New minor version of npm available! 7.7.6 -> 7.19.0
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v7.19.0>
npm notice Run `npm install -g npm@7.19.0` to update!
npm notice
The discord bot should be able to use the public external hostname, but it would be preferrable to use a service of type ClusterIP for intra cluster communication (https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types)
Thank you 👍 I will have a play 🤓
I ssh onto the pod and it can ping the API, so not sure why when node does it, it then fails 🤔
Bot -> API
# ping api.eddiehub.org
PING api.eddiehub.org (67.207.70.148) 56(84) bytes of data.
64 bytes from ingress-nginx-controller.ingress-nginx.svc.cluster.local (67.207.70.148): icmp_seq=1 ttl=58 time=2.48 ms
64 bytes from ingress-nginx-controller.ingress-nginx.svc.cluster.local (67.207.70.148): icmp_seq=2 ttl=58 time=1.16 ms
64 bytes from ingress-nginx-controller.ingress-nginx.svc.cluster.local (67.207.70.148): icmp_seq=3 ttl=58 time=0.794 ms
I tried so many things, I got it working, but maybe this is a stupid move, I ssh'd on to the server and wrote a small js script to test all the options out. I got it to work with the service name, is that allowed? 🤷♂️
require('http').get('http://eddiehub-api-service:3000', function(res) { console.log(res.statusCode) }).on('error', console.log)
Using the service name is the correct way to do it: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
If you are in the same namespace you can use just the service name:
http://eddiehub-api-service:3000
If the discord bot were in a different namespace you would need to specify the namespace of the eddiehub API service:
http://eddiehub-api-service.<OTHER-NAMESPACE>:3000
or if you wanted you could use the fully qualified domain name of the service:
http://eddiehub-api-service.default.svc.cluster.local:3000
ok great, thank you for clarifying 👍 .
I added a new environment variable to the Kubernetes deployment, but now it won't deploy Error: RolloutStatusTimedout
, not sure what I broke. I will take a look in the morning.
20
deployment.apps/eddiebot-deployment unchanged
21
/opt/hostedtoolcache/kubectl/1.21.2/x64/kubectl rollout status Deployment/eddiebot-deployment --namespace default
22
error: deployment "eddiebot-deployment" exceeded its progress deadline
23
Error: Error: error: deployment "eddiebot-deployment" exceeded its progress deadline
24
/opt/hostedtoolcache/kubectl/1.21.2/x64/kubectl describe Deployment eddiebot-deployment --namespace default
25
Name: eddiebot-deployment
26
Namespace: default
27
CreationTimestamp: Sat, 15 May 2021 16:24:35 +0000
28
Labels: app=eddiebot
29
workflow=githubWorkflow_f95061c0b2fad0e256bf3d640aa60f85
30
workflowFriendlyName=Publish_Docker_image_and_deploy
31
Annotations: deployment.kubernetes.io/revision: 21
32
githubWorkflow_f95061c0b2fad0e256bf3d640aa60f85:
33
{'run': '994162677','repository': 'EddieHubCommunity/EddieBot','workflow': 'Publish Docker image and deploy','workflowFileName': 'docker.y...
34
Selector: app=eddiebot
35
Replicas: 1 desired | 1 updated | 2 total | 1 available | 1 unavailable
36
StrategyType: RollingUpdate
37
MinReadySeconds: 0
38
RollingUpdateStrategy: 25% max unavailable, 25% max surge
39
Pod Template:
40
Labels: app=eddiebot
41
Annotations: kubectl.kubernetes.io/restartedAt: 2021-05-15T17:46:44+01:00
42
Containers:
43
eddiebot:
44
Image: ghcr.io/eddiehubcommunity/eddiebot:v0.10.1
45
Port: <none>
46
Host Port: <none>
47
Environment:
48
DISCORD_TOKEN: <set to the key 'DISCORD_TOKEN' in secret 'discord-token'> Optional: false
49
DISCORD_PREFIX: <set to the key 'DISCORD_PREFIX' in secret 'discord-prefix'> Optional: false
50
API_URL: <set to the key 'API_URL' in secret 'api-url'> Optional: false
51
Mounts: <none>
52
Volumes: <none>
53
Conditions:
54
Type Status Reason
55
---- ------ ------
56
Available True MinimumReplicasAvailable
57
Progressing False ProgressDeadlineExceeded
58
OldReplicaSets: eddiebot-deployment-748645ccc9 (1/1 replicas created)
59
NewReplicaSet: eddiebot-deployment-798b5cc95d (1/1 replicas created)
60
Events:
61
Type Reason Age From Message
62
---- ------ ---- ---- -------
63
Normal ScalingReplicaSet 14m deployment-controller Scaled up replica set eddiebot-deployment-798b5cc95d to 1
64
Error: Error: RolloutStatusTimedout
https://github.com/EddieHubCommunity/EddieBot/runs/2980760549?check_suite_focus=true
Not sure what the error is, but also you don't need to put things like the API URL into a secret (I noticed <set to the key 'API_URL' in secret 'api-url'>
)
Those can be directly in the discord bot deployment YAML since it is not sensitive. Adding the secret creates another layer of unnecessary complexity for things that are not actually secret.
You are right, it is not a secret. I kept it the same as the others to try and keep things consistent. I did add it to the GitHub secrets and the deployment yaml https://github.com/EddieHubCommunity/EddieBot/commit/a0b1e240bf13a388d8ce07aefc99e4db8e15f7dc
I think that was everywhere I needed to add it
It doesn't like my new config Error: secret "api-url" not found
NAME READY STATUS RESTARTS AGE
eddiebot-deployment-748645ccc9-z8hwf 1/1 Running 1 31h
eddiebot-deployment-798b5cc95d-zpmp7 0/1 CreateContainerConfigError 0 73m
eddiehub-api-deployment-6d7b8778c8-d9jz5 1/1 Running 0 4d8h
eddiehub-api-deployment-6d7b8778c8-rx5mr 1/1 Running 0 4d8h
stargate-deployment-5dfdfbbf58-66nnp 1/1 Running 0 60d
➜ Downloads kubectl describe pod eddiebot-deployment-798b5cc95d-zpmp7
Name: eddiebot-deployment-798b5cc95d-zpmp7
Namespace: default
Priority: 0
Node: pool-ozh49d95j-87v3s/10.131.25.238
Start Time: Sat, 03 Jul 2021 23:01:03 +0100
Labels: app=eddiebot
pod-template-hash=798b5cc95d
Annotations: kubectl.kubernetes.io/restartedAt: 2021-05-15T17:46:44+01:00
Status: Pending
IP: 10.244.0.32
IPs:
IP: 10.244.0.32
Controlled By: ReplicaSet/eddiebot-deployment-798b5cc95d
Containers:
eddiebot:
Container ID:
Image: ghcr.io/eddiehubcommunity/eddiebot:v0.10.1
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: CreateContainerConfigError
Ready: False
Restart Count: 0
Environment:
DISCORD_TOKEN: <set to the key 'DISCORD_TOKEN' in secret 'discord-token'> Optional: false
DISCORD_PREFIX: <set to the key 'DISCORD_PREFIX' in secret 'discord-prefix'> Optional: false
API_URL: <set to the key 'API_URL' in secret 'api-url'> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-gbz7w (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-gbz7w:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-gbz7w
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Failed 24m (x247 over 79m) kubelet Error: secret "api-url" not found
Normal Pulling 4m11s (x336 over 79m) kubelet Pulling image "ghcr.io/eddiehubcommunity/eddiebot:v0.10.1"
Oh I forgot to let Kubernetes know about the secret 🤦♂️
I forgot to do this
export TMP_SECRET_YAML_FILE=secret.yaml
kubectl create secret generic discord-bot-secret --from-literal=DISCORD_BOT_TOKEN=${{ secrets.DISCORD_BOT_TOKEN }} --dry-run -o yaml > $TMP_SECRET_YAML_FILE
kubectl apply -f secret.yaml
rm $TMP_SECRET_YAML_FILE
Ok it works now 🎉
Thanks @sidpalas 👍 - you make it all look so easy when we are pairing 😎
Need to check Kubernetes logs to understand why it causes the bot to crash (works locally with the prod API)