Closed caoluong2k closed 1 year ago
sepfy
commented
1 year ago hi, did you run the example on x86 platform? if yes, you need to change the gstreamer pipeline v4l2src ! video/x-raw,width=1920,height=1072,framerate=30/1 ! omxh264enc !. it's for raspberry pi
caoluong2k
commented
1 year ago Hi @sepfy i run the example on x86 platform. But I am not clear about gstreamer. How do I modify const char PIPELINE[ ] to ? Thank you very much. Sorry for bothering you.
sepfy
commented
1 year ago what video you want to show? is it webcam? could you try to change the pipeline of surveillance example to:
v4l2src ! video/x-raw,width=640,height=480,framerate=30/1 ! videoconvert ! queue ! x264enc bitrate=6000 speed-preset=ultrafast tune=zerolatency key-int-max=15 ! video/x-h264,profile=constrained-baseline ! rtph264pay name=rtp config-interval=-1 ssrc=1 ! appsink name=peer-connection-sinkor from test source
videotestsrc ! video/x-raw,width=640,height=480,framerate=30/1 ! videoconvert ! queue ! x264enc bitrate=6000 speed-preset=ultrafast tune=zerolatency key-int-max=15 ! video/x-h264,profile=constrained-baseline ! rtph264pay name=rtp config-interval=-1 ssrc=1 ! appsink name=peer-connection-sinkAnd you need to ensure that the libxh264 is installed, because video content is encoded as h264
caoluong2k
commented
1 year ago Thank you very much. I wish you the best
hialle
commented
1 year ago hello @sepfy, I run example homecamera example, but it not play video and audio on chrome. What do I need to convert audio PIPELINE[ ] to if I run on x86 PC ?
sepfy
commented
1 year ago @hialle for video, you can try this:
v4l2src ! video/x-raw,width=640,height=480,framerate=30/1 ! videoconvert ! queue ! x264enc bitrate=6000 speed-preset=ultrafast tune=zerolatency key-int-max=15 ! video/x-h264,profile=constrained-baseline ! rtph264pay name=rtp config-interval=-1 ssrc=1 ! appsink name=peer-connection-sink
and run GST_DEBUG=2 ./home_camera to capture more log.
hialle
commented
1 year ago Hi @sepfy I want integrate TCP signaling server for surveilance example.
My idea is, camera will listen for SDP offer from website via TCP server and when camera receive offer I will call signaling_observer_notify_event(signaling_http->signaling_observer, SIGNALING_EVENT_GET_OFFER, SDP_Offer) and oncallevent() function. But the program stops when the oncallevent() and signaling_observer_notify_event is done. Do you have any suggestions?
Sorry for bothering you
sepfy
commented
1 year ago hello ~@hialle, how do you implement the oncallevent() function ? when receive the SDP, the camera need to response the answer. in the surveilance example, the answer should be obtained after this function call peer_connection_set_remote_description. you can obtain the answer in the callback function on_icecandidate , then you can send the answer to the TCP server
hialle
commented
1 year ago Thanks for your help @sepfy . Your project has helped many students like me understand WebRTC.
Here is the on_call_event() function in file main:
void on_call_event(SignalingEvent signaling_event, char *msg, void *data)
{
LOG_INFO("----------> On call event START");
if (signaling_event == SIGNALING_EVENT_GET_OFFER)
{
printf("Get offer from singaling\n");
g_mutex_lock(&g_surveillance.mutex);
gst_element_set_state(g_surveillance.pipeline, GST_STATE_PAUSED);
peer_connection_destroy(g_surveillance.pc);
g_surveillance.pc = peer_connection_create();
MediaStream *media_stream = media_stream_new();
media_stream_add_track(media_stream, CODEC_H264);
peer_connection_add_stream(g_surveillance.pc, media_stream);
peer_connection_create_answer(g_surveillance.pc); // ---> nice agent gather candidate
peer_connection_set_remote_description(g_surveillance.pc, msg); // NCL
peer_connection_onicecandidate(g_surveillance.pc, on_icecandidate, NULL); // ---> Send Answer
peer_connection_oniceconnectionstatechange(g_surveillance.pc, &on_iceconnectionstatechange, NULL);
peer_connection_on_connected(g_surveillance.pc, on_connected, NULL);
peer_connection_on_receiver_packet_loss(g_surveillance.pc, on_receiver_packet_loss, NULL);
g_cond_wait(&g_surveillance.cond, &g_surveillance.mutex);
g_mutex_unlock(&g_surveillance.mutex);
}
LOG_INFO("-----------> On call event END");
}And in signaling_http.c file, I create two function:
#if 1
struct addrinfo hints;
struct addrinfo *signaling_server_address;
SOCKET socket_peer_tmp;
SignalingHttp *signaling_server_create(const char *host, int port,
SignalingObserver *signaling_observer)
{
client_init(host, port);
SignalingHttp *signaling_server = (SignalingHttp *)malloc(sizeof(SignalingHttp));
if (!signaling_server)
{
return NULL;
}
signaling_server->answer = NULL;
signaling_server->socket_peer = socket_peer_tmp;
signaling_server->signaling_observer = signaling_observer;
return signaling_server;
}
void signaling_send_answer(SignalingHttp *signaling_http, char *sdp)
{
printf("------> Signaling send answer\n");
char answer[SDP_MAX_SIZE];
snprintf(answer, SDP_MAX_SIZE, "{\"type\": \"answer\", \"sdp\": \"%s\"}", sdp);
send(signaling_http->socket_peer, (char *)answer, strlen(answer), 0);
signaling_http->answer = sdp;
LOG_INFO(" =======> CAMERA send SDP to SIGNALING SERVER:\n %s", answer);
LOG_INFO("\n%li (bytes)\n", sizeof(answer));
memset(answer, 0, SDP_MAX_SIZE);
}
void signaling_listen(SignalingHttp *signaling_http)
{
char buf_sdp[SDP_MAX_SIZE];
int n = 0;
LOG_INFO("Wait SDP msg from Server ....... \n");
n = recv(signaling_http->socket_peer, (char *)buf_sdp, sizeof(buf_sdp), 0);
printf("CAMERA received (%d bytess) SDP from WEB through SIGNALING SERVER: \n%.*s \n",
n, n, buf_sdp);
do
{
if (n > 0)
{
signaling_observer_notify_event(signaling_http->signaling_observer, SIGNALING_EVENT_GET_OFFER,
buf_sdp);
}
} while (0);
memset(buf_sdp, 0, SDP_MAX_SIZE);
printf("========> memset\n");
}
void client_init(char *host, uint16_t port)
{
memset(&hints, 0, sizeof(hints));
hints.ai_socktype = SOCK_STREAM;
// int to char
char port_convert[5];
int num = port;
sprintf(port_convert, "%i", num);
if (getaddrinfo(host, port_convert, &hints, &signaling_server_address))
{
fprintf(stderr, "getaddrinfo() failed. (%d)\n", GETSOCKETERRNO());
}
printf("Create socket ...\n");
socket_peer_tmp = socket(signaling_server_address->ai_family,
signaling_server_address->ai_socktype, signaling_server_address->ai_protocol);
if (!ISVALIDSOCKET(socket_peer_tmp))
{
fprintf(stderr, "socket() failed. (%d)\n", GETSOCKETERRNO());
}
printf("Connecting to siganling server...\n");
if (connect(socket_peer_tmp, signaling_server_address->ai_addr, signaling_server_address->ai_addrlen))
{
fprintf(stderr, "connect() failed. (%d)\n", GETSOCKETERRNO());
}
freeaddrinfo(signaling_server_address);
printf("Connected.\n");
// while (1)
//{
fd_set reads;
FD_ZERO(&reads);
FD_SET(socket_peer_tmp, &reads);
struct timeval timeout;
timeout.tv_sec = 0;
timeout.tv_usec = 100000;
if (select(socket_peer_tmp + 1, &reads, 0, 0, &timeout) < 0)
{ // use select() for terminal input
fprintf(stderr, "select() failed. (%d)\n", GETSOCKETERRNO());
}
}
#endifAnd signaling.c file, I add more function:
Signaling *signaling_create(SignalingOption signaling_option)
{
LOG_INFO("Signaling Create");
Signaling *signaling = (Signaling *)malloc(sizeof(Signaling));
gboolean impl_created = FALSE;
if (!signaling)
return NULL;
signaling->option = signaling_option;
switch (signaling->option.protocol)
{
case SIGNALING_PROTOCOL_HTTP:
signaling->impl = (void *)signaling_http_create(signaling->option.host, signaling->option.port,
signaling->option.call, signaling->option.index_html,
&signaling->observer);
break;
default:
signaling->impl = (void *)signaling_server_create(signaling->option.host, signaling->option.port,
&signaling->observer);
break;
}
if (!signaling->impl)
{
free(signaling);
return NULL;
}
return signaling;
}
void signaling_dispatch(Signaling *signaling)
{
LOG_INFO("Signaling dispatch");
switch (signaling->option.protocol)
{
case SIGNALING_PROTOCOL_HTTP:
signaling_http_dispatch((SignalingHttp *)signaling->impl);
break;
default:
signaling_listen((SignalingHttp *)signaling->impl);
break;
}
}
void signaling_send_answer_to_call(Signaling *signaling, char *sdp)
{
LOG_INFO("Signaling send Answer to Call");
switch (signaling->option.protocol)
{
case SIGNALING_PROTOCOL_HTTP:
signaling_http_set_answer((SignalingHttp *)signaling->impl, sdp);
break;
default:
signaling_send_answer((SignalingHttp *)signaling->impl, sdp);
break;
}
}I'm glad to help you. could you check signaling_send_answer and LOG_INFO("-----------> On call event END"); is called? if On call event END is not called, maybe there is some issue in offer. And could you use the latest version of library? because i merge some code recently
hialle
commented
1 year ago Hi @sepfy , I use the latest version of library. And On call event END is called.
Hi @sepfy @aldoshkind @icemanaf @timgates42 could you help me ?. Your help will help me a lot at this time.
I successfully run video on demand example but with surveillance and home_camera examples I can't play video and audio on chorme. The P2P has been established. we can see the data can be sent from device to chrome from wireshark but the video and audio doesn't display on chrome. Even though The P2P has been established. I can see the data can be sent from device to chrome from wireshark but the video and audio doesn't display on chrome. (I fixed h264 codec to 102). (In home_camera, I only see audio data).
This is SDP Offer when I run _Homecamera example:
v=0 (6 more lines) o=- 499682685197050615 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE 0 1 a=extmap-allow-mixed a=msid-semantic: WMS m=audio 9 UDP/TLS/RTP/SAVPF 111 63 103 104 9 0 8 106 105 13 110 112 113 126 (34 more lines) mid=0 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:f++w a=ice-pwd:pJSBZ8xFxWCG1p5Ue4IZDCdL a=ice-options:trickle a=fingerprint:sha-256 AB:5A:07:9C:72:2E:EB:C0:2B:04:39:C5:FC:AA:37:3C:BF:A3:C6:63:45:E0:11:4D:7E:0D:B7:B6:42:64:B0:B7 a=setup:actpass a=mid:0 a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid a=sendrecv a=msid:- c945ed4d-d989-41c0-b5f8-ca4369406ee7 a=rtcp-mux a=rtpmap:111 opus/48000/2 a=rtcp-fb:111 transport-cc a=fmtp:111 minptime=10;useinbandfec=1 a=rtpmap:63 red/48000/2 a=fmtp:63 111/111 a=rtpmap:103 ISAC/16000 a=rtpmap:104 ISAC/32000 a=rtpmap:9 G722/8000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:106 CN/32000 a=rtpmap:105 CN/16000 a=rtpmap:13 CN/8000 a=rtpmap:110 telephone-event/48000 a=rtpmap:112 telephone-event/32000 a=rtpmap:113 telephone-event/16000 a=rtpmap:126 telephone-event/8000 a=ssrc:1238015745 cname:LFwx9XZ3VFIUxAJx a=ssrc:1238015745 msid:- c945ed4d-d989-41c0-b5f8-ca4369406ee7 m=video 9 UDP/TLS/RTP/SAVPF 96 97 102 123 127 122 125 107 108 109 124 121 39 40 45 46 98 99 100 101 120 119 114 (120 more lines) mid=1 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:f++w a=ice-pwd:pJSBZ8xFxWCG1p5Ue4IZDCdL a=ice-options:trickle a=fingerprint:sha-256 AB:5A:07:9C:72:2E:EB:C0:2B:04:39:C5:FC:AA:37:3C:BF:A3:C6:63:45:E0:11:4D:7E:0D:B7:B6:42:64:B0:B7 a=setup:actpass a=mid:1 a=extmap:14 urn:ietf:params:rtp-hdrext:toffset a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:13 urn:3gpp:video-orientation a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id a=sendrecv a=msid:- e07a6190-2e38-4c89-a2a6-123a7ddec806 a=rtcp-mux a=rtcp-rsize a=rtpmap:96 VP8/90000 a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtpmap:97 rtx/90000 a=fmtp:97 apt=96 a=rtpmap:102 H264/90000 a=rtcp-fb:102 goog-remb a=rtcp-fb:102 transport-cc a=rtcp-fb:102 ccm fir a=rtcp-fb:102 nack a=rtcp-fb:102 nack pli a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f a=rtpmap:123 rtx/90000 a=fmtp:123 apt=102 a=rtpmap:127 H264/90000 a=rtcp-fb:127 goog-remb a=rtcp-fb:127 transport-cc a=rtcp-fb:127 ccm fir a=rtcp-fb:127 nack a=rtcp-fb:127 nack pli a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f a=rtpmap:122 rtx/90000 a=fmtp:122 apt=127 a=rtpmap:125 H264/90000 a=rtcp-fb:125 goog-remb a=rtcp-fb:125 transport-cc a=rtcp-fb:125 ccm fir a=rtcp-fb:125 nack a=rtcp-fb:125 nack pli a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f a=rtpmap:107 rtx/90000 a=fmtp:107 apt=125 a=rtpmap:108 H264/90000 a=rtcp-fb:108 goog-remb a=rtcp-fb:108 transport-cc a=rtcp-fb:108 ccm fir a=rtcp-fb:108 nack a=rtcp-fb:108 nack pli a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f a=rtpmap:109 rtx/90000 a=fmtp:109 apt=108 a=rtpmap:124 H264/90000 a=rtcp-fb:124 goog-remb a=rtcp-fb:124 transport-cc a=rtcp-fb:124 ccm fir a=rtcp-fb:124 nack a=rtcp-fb:124 nack pli a=fmtp:124 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d001f a=rtpmap:121 rtx/90000 a=fmtp:121 apt=124 a=rtpmap:39 H264/90000 a=rtcp-fb:39 goog-remb a=rtcp-fb:39 transport-cc a=rtcp-fb:39 ccm fir a=rtcp-fb:39 nack a=rtcp-fb:39 nack pli a=fmtp:39 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=4d001f a=rtpmap:40 rtx/90000 a=fmtp:40 apt=39 a=rtpmap:45 AV1/90000 a=rtcp-fb:45 goog-remb a=rtcp-fb:45 transport-cc a=rtcp-fb:45 ccm fir a=rtcp-fb:45 nack a=rtcp-fb:45 nack pli a=rtpmap:46 rtx/90000 a=fmtp:46 apt=45 a=rtpmap:98 VP9/90000 a=rtcp-fb:98 goog-remb a=rtcp-fb:98 transport-cc a=rtcp-fb:98 ccm fir a=rtcp-fb:98 nack a=rtcp-fb:98 nack pli a=fmtp:98 profile-id=0 a=rtpmap:99 rtx/90000 a=fmtp:99 apt=98 a=rtpmap:100 VP9/90000 a=rtcp-fb:100 goog-remb a=rtcp-fb:100 transport-cc a=rtcp-fb:100 ccm fir a=rtcp-fb:100 nack a=rtcp-fb:100 nack pli a=fmtp:100 profile-id=2 a=rtpmap:101 rtx/90000 a=fmtp:101 apt=100 a=rtpmap:120 red/90000 a=rtpmap:119 rtx/90000 a=fmtp:119 apt=120 a=rtpmap:114 ulpfec/90000 a=ssrc-group:FID 2552703379 1944201314 a=ssrc:2552703379 cname:LFwx9XZ3VFIUxAJx a=ssrc:2552703379 msid:- e07a6190-2e38-4c89-a2a6-123a7ddec806 a=ssrc:1944201314 cname:LFwx9XZ3VFIUxAJx a=ssrc:1944201314 msid:- e07a6190-2e38-4c89-a2a6-123a7ddec806
This is SDP Answer when I run _Homecamera example:
v=0 (5 more lines) o=- 2023662770 2023662770 IN IP4 0.0.0.0 s=- t=0 0 a=msid-semantic: WMS a=group:BUNDLE 0 1 m=audio 9 UDP/TLS/RTP/SAVP 8 (17 more lines) mid=0 a=rtpmap:8 PCMA/8000 a=ssrc:2 cname:pear c=IN IP4 0.0.0.0 a=sendrecv a=mid:0 a=rtcp-mux a=ice-ufrag:5iV9 a=ice-pwd:iS1uFIj3df0oUBVG2HUcsF a=ice-options:trickle a=fingerprint:sha-256 87:EC:85:7F:EA:36:C4:40:59:DE:40:BD:EE:D0:46:31:51:20:A5:78:97:7B:32:8E:70:DD:04:F6:38:DE:3B:A5 a=setup:passive a=candidate:1 1 udp 2015363327 192.168.50.154 38159 typ host a=candidate:2 1 udp 1015021823 192.168.50.154 0 typ host a=candidate:3 1 udp 1010827519 192.168.50.154 49283 typ host a=candidate:7 1 udp 1679819007 116.104.141.202 5572 typ host a=candidate:8 1 udp 847249663 116.104.141.202 0 typ host a=candidate:9 1 udp 843055359 116.104.141.202 49283 typ host m=video 9 UDP/TLS/RTP/SAVPF 96 102 (23 more lines) mid=1 a=rtcp-fb:102 nack a=rtcp-fb:102 nack pli a=fmtp:96 profile-level-id=42e01f;level-asymmetry-allowed=1 a=fmtp:102 profile-level-id=42e01f;packetization-mode=1;level-asymmetry-allowed=1 a=fmtp:102 x-google-max-bitrate=6000;x-google-min-bitrate=2000;x-google-start-bitrate=4000 a=rtpmap:96 H264/90000 a=rtpmap:102 H264/90000 a=ssrc:1 cname:pear c=IN IP4 0.0.0.0 a=sendrecv a=mid:1 a=rtcp-mux a=ice-ufrag:5iV9 a=ice-pwd:iS1uFIj3df0oUBVG2HUcsF a=ice-options:trickle a=fingerprint:sha-256 87:EC:85:7F:EA:36:C4:40:59:DE:40:BD:EE:D0:46:31:51:20:A5:78:97:7B:32:8E:70:DD:04:F6:38:DE:3B:A5 a=setup:passive a=candidate:1 1 udp 2015363327 192.168.50.154 38159 typ host a=candidate:2 1 udp 1015021823 192.168.50.154 0 typ host a=candidate:3 1 udp 1010827519 192.168.50.154 49283 typ host a=candidate:7 1 udp 1679819007 116.104.141.202 5572 typ host a=candidate:8 1 udp 847249663 116.104.141.202 0 typ host a=candidate:9 1 udp 843055359 116.104.141.202 49283 typ host
Stats Table