cedricve
commented
3 years ago Thanks @andrewheberle from the logs you can see your camera is rebooting continuously. What type of ip camera are you using?
Closed andrewheberle closed 2 years ago
cedricve
commented
3 years ago Thanks @andrewheberle from the logs you can see your camera is rebooting continuously. What type of ip camera are you using?
andrewheberle
commented
3 years ago Hi @cedricve i don’t think that is the case as the IP camera in question is showing an uptime of 13 days.
The camera is a Ubiquiti UVC-G2 camera that is set up in “standalone” mode so it’s just a “dumb” camera with a 720p RTSP stream (ie not using their Video setup).
This works fine with the open source/docker setup and my other two non-enterprise agents are still running fine with two other cameras of the same model.
andrewheberle
commented
3 years ago Just to give some more context, I had made a bunch of config changes during the period of those logs to try and get things working so I’m not sure if that has “polluted” the logs a bit.
cedricve
commented
3 years ago hmm yeah it looks like it can connect, but it fails after x seconds (or you reconfigured).
Can you share a longer dump using following command
kunectl logs machiner-....
Please attach as a txt file.
andrewheberle
commented
3 years ago Here are the full logs from last night.
It definitely looks like its failing to read the stream properly, which is unusual as this is the same camera and also on the same physical node/hardware that was running the open source Docker based machinery with no problems (this container is not running anymore).
cedricve
commented
3 years ago The enterprise is a complete rewrite so the behaviour might differ, will have a look.
cedricve
commented
3 years ago Interesting looks like you have audio enabled. Might be a bug on our side. Could you already try to disable the audio on your IP camera?
00:16:55.886 Bootstrap ▶ INFO db22[0m Opening RTSP stream
00:16:55.936 WriteToTrack ▶ INFO db23[0m WEBRTC: listing codecs.
00:16:55.936 RecordStream ▶ INFO db24[0m Start recording routine
00:16:55.936 ProcessMotion ▶ INFO db25[0m Start Detecting motion...
00:16:55.936 WriteToTrack ▶ INFO db26[0m WEBRTC: codec - AAC found.
00:16:55.936 WriteToTrack ▶ INFO db27[0m {[21 8] {11025 1ch 2 10 1}}
00:16:55.936 RecordStream ▶ INFO db28[0m Start motion based recording
00:16:55.936 WriteToTrack ▶ INFO db29[0m WEBRTC: codec - H264 found.Not directly unfortunately.
These cameras allow me to turn off the microphone but looking at the RTSP stream in VLC the audio stream still exists.
As a workaround to test the theory I’ll deploy a ffmpeg or VLC container to restream the camera without audio and let you know.
cedricve
commented
3 years ago Yeah will look on our side, I expect a bug as the audio is coming first before the h264 codec. Will verify and create a new release.
cedricve
commented
3 years ago @andrewheberle can you give 1.3.17 a try?
andrewheberle
commented
3 years ago @cedricve unfortunately 1.3.17 behaves the same and I have not yet had a chance to set up a service to re-stream video only.
The logs from the machinery after the 1.3.17 update are attached if they help (although they look basically the same to me):
cedricve
commented
3 years ago ok thanks for testing @andrewheberle, let me know when you would have time to look into it together, would love to resolve the issue!
andrewheberle
commented
3 years ago Sorry @cedricve I was not totally correct. The uploaded videos now show a preview image but so far all but one uploaded videos are still unwatchable and apparently 4-seconds long.
I have set up a "re-streamer" using VLC using the following command line in an effort to strip out the audio channel:
vlc.exe rtsp://192.168.11.20:554/s0 --no-sout-audio --sout='#rtp{sdp=rtsp://:8554/}'
Unfortunately this results in the machinery going into a crash/restart loop:
2021-01-11 13:56:40,936 INFO spawned: 'machinery' with pid 14
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)
[GIN-debug] GET /debug/pprof/ --> github.com/gin-contrib/pprof.pprofHandler.func1 (3 handlers)
[GIN-debug] GET /debug/pprof/cmdline --> github.com/gin-contrib/pprof.pprofHandler.func1 (3 handlers)
[GIN-debug] GET /debug/pprof/profile --> github.com/gin-contrib/pprof.pprofHandler.func1 (3 handlers)
[GIN-debug] POST /debug/pprof/symbol --> github.com/gin-contrib/pprof.pprofHandler.func1 (3 handlers)
[GIN-debug] GET /debug/pprof/symbol --> github.com/gin-contrib/pprof.pprofHandler.func1 (3 handlers)
[GIN-debug] GET /debug/pprof/trace --> github.com/gin-contrib/pprof.pprofHandler.func1 (3 handlers)
[GIN-debug] GET /debug/pprof/allocs --> github.com/gin-contrib/pprof.pprofHandler.func1 (3 handlers)
[GIN-debug] GET /debug/pprof/block --> github.com/gin-contrib/pprof.pprofHandler.func1 (3 handlers)
[GIN-debug] GET /debug/pprof/goroutine --> github.com/gin-contrib/pprof.pprofHandler.func1 (3 handlers)
[GIN-debug] GET /debug/pprof/heap --> github.com/gin-contrib/pprof.pprofHandler.func1 (3 handlers)
[GIN-debug] GET /debug/pprof/mutex --> github.com/gin-contrib/pprof.pprofHandler.func1 (3 handlers)
[GIN-debug] GET /debug/pprof/threadcreate --> github.com/gin-contrib/pprof.pprofHandler.func1 (3 handlers)
13:56:41.127 Bootstrap ▶ INFO 001 Opening configuration
[GIN-debug] GET /restart --> main.main.func6 (4 handlers)
[GIN-debug] GET /motion --> main.main.func7 (4 handlers)
[GIN-debug] GET /config --> main.main.func8 (5 handlers)
[GIN-debug] POST /config --> main.main.func9 (5 handlers)
[GIN-debug] Listening and serving HTTP on :8080
13:56:41.130 Bootstrap ▶ INFO 002 Opened configurations
13:56:41.130 Bootstrap ▶ INFO 003 Creating livestream and webrtc channel
13:56:41.130 Bootstrap ▶ INFO 004 Starting configuration MQTT
13:56:41.804 Bootstrap ▶ INFO 005 Creating packet buffers
13:56:41.804 func1 ▶ INFO 006 MQTT 24: connected to tcp://mqtt.kerberos.io:1883
13:56:41.806 Bootstrap ▶ INFO 007 Opening RTSP stream
13:56:41.807 func1 ▶ INFO 008 MQTT: sending logs to kerberos/AKIA5V6EGLUW4UDVIY5L/device/cE4Tn0aVx0OjySOzraepmfYb9ulJZU/request-live
13:56:41.904 ProcessMotion ▶ INFO 009 Start Detecting motion...
13:56:41.904 RecordStream ▶ INFO 00a Start recording routine
13:56:41.906 RecordStream ▶ INFO 00c Start motion based recording
13:56:41.904 WriteToTrack ▶ INFO 00b WEBRTC: listing codecs.
13:56:41.906 WriteToTrack ▶ INFO 00d WEBRTC: codec - H264 found.
13:56:41.906 WriteToTrack ▶ INFO 00e {[1 77 0 31 255 225 0 24 103 77 0 31 154 102 2 128 45 255 128 183 1 1 1 64 0 0 250 0 0 58 152 37 1 0 4 104 238 60 128] {77 0 31 3 [[103 77 0 31 154 102 2 128 45 255 128 183 1 1 1 64 0 0 250 0 0 58 152 37]] [[104 238 60 128]]} {77 31 80 45 0 0 0 0 1280 720 60000 2000}}
13:56:41.906 WriteToTrack ▶ INFO 00f WEBRTC: not using a transcoder.
2021-01-11 13:56:42,908 INFO success: machinery entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
13:56:56.905 Stop ▶ INFO 010 Stopping stream...
13:56:56.906 Stop ▶ INFO 011 Stopping upload...
13:56:56.906 Stop ▶ INFO 012 Stopping sendalive...
13:56:56.906 Stop ▶ INFO 013 Stopping motion...
13:56:56.906 Stop ▶ INFO 014 Stopping recording...
13:56:56.906 Stop ▶ INFO 015 Stopping livestream...
13:56:56.906 Stop ▶ INFO 016 Stopping RTSP stream...
13:56:56.907 HandleStream ▶ ERRO 017 read tcp 10.42.0.111:52246->192.168.12.73:8554: use of closed network connection
13:56:56.907 Stop ▶ INFO 018 Stopping channels...
13:56:56.907 Stop ▶ INFO 019 Stopping MQTT client
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xa322c2]
goroutine 79 [running]:
gocv.io/x/gocv.(*Mat).Close.func1(0x0)
/go/pkg/mod/gocv.io/x/gocv@v0.25.0/mat_noprofile.go:18 +0x22
gocv.io/x/gocv.(*Mat).Close(0x0, 0xc000521580, 0x1)
/go/pkg/mod/gocv.io/x/gocv@v0.25.0/mat_noprofile.go:18 +0x2b
gitlab.com/kerberos-io/machinery/src/capture.ProcessMotion(0xc0002def90, 0xc00003d140, 0x6, 0xc00003aaa0, 0x1e, 0xc00003d080, 0x5, 0xc00003d020, 0xf, 0xc00003d070, ...)
/go/src/gitlab.com/kerberos-io/factory/machinery/src/capture/ImageProcessing.go:176 +0xf05
created by main.Bootstrap
/go/src/gitlab.com/kerberos-io/factory/machinery/main.go:83 +0x9fe
2021-01-11 13:56:56,920 INFO exited: machinery (exit status 2; not expected)My "re-stream" can be played via another VLC instance on another device locally and shows the video channel only as shown below:
If I can provide any more info to help let me know.
cedricve
commented
3 years ago @andrewheberle let's jump on a call, would like to see it in action. Regards Cédric
cedricve
commented
3 years ago I know also see, it sometimes work (as it can do motion detection). But afterwards receives a bad packet EOF. VLC is more error prone and will ignore these packets I assume. Do you have some encoding settings you can tweak?
12:03:15.390 FindMotion ▶ INFO 8c1c[0m Number of changes detected:18
12:03:17.233 WriteToTrack ▶ INFO 8c1d[0m WEBRTC: Sending keyframe
12:03:17.407 FindMotion ▶ INFO 8c1e[0m Number of changes detected:0
[31m12:03:17.538 HandleStream ▶ ERRO 8c1f[0m EOF
WEBRTC: stop writing to track.Sorry for the late response.
Unfortunately I don’t have any parameters I can tweak on these cameras apart from the two different resolution streams I can access (both have issues).
I’m happy to have a call so you can see first hand.
I’m in GMT+8 so if there is an overlap between your own timezone we can sort out a call/screen share.
cedricve
commented
2 years ago By now should have been fixed, major changes have been added since this issue post.
I have three agents deployed that have been working well (using Kerberos Opensource under Docker) for the last 12 days (new Kerberos user) and have just migrated one agent to Kerberos Enterprise today.
Versions in user are:
Factory = 1.4.5 Machinery = 1.3.16
This camera shows up fine in Kerberos Cloud however no uploaded videos play properly, with them seemingly being a few seconds long with no audio or video.
I have tried playing these videos (grabbed from app.kerberos.io) using VLC with no success.
Live streaming of this camera does work and shows images, however only in Low Res, which I would have expected Full Res to be available as this is a Kerberos Enterprise agent.
Configuring zones from the Kerberos Factory for this camera works fine too (ie the stream is visible) so my newly deployed machinery seems to be picking up the stream from the camera fine.
The following is the logs from the machinery pod: