Test Ideas - Githubissues

sipsorcery / webrtc-echoes

Simple useful interoperability tests for WebRTC libraries. If you are a WebRTC library developer we'd love to include you!

171 stars 17 forks source link

Test Ideas #7

Open Sean-Der opened 3 years ago

Sean-Der commented 3 years ago

Echo (Round trip audio/video packets)

Signaling/Media

Renegotation
Simulcast
Multiple media sections
Offerer controls PayloadTypes
Answerer properly responds to enabled/disabled codecs

ICE

pwd/ufrag validity is assert
Trickle
Restart
STUN
TURN (UDP, TCP, DTLS and TLS)
mDNS

DTLS

assert that fingerprint is checked

DataChannels

Ordered and Unordered
Lossy and Lossless
Assert IDs are properly chosen

RTP/RTCP

Sender/Receiver Reports
Responds to NACK

PeerConnection APIs

ICEConnectionState, PeerConnectionState

Sean-Der commented 3 years ago

@sipsorcery @shinyoshiaki @steely-glint @jlaine what do you think? Some of these come from bugs/mistakes in Pion that really hurt, and I don't want to see others have to deal with.

sipsorcery commented 3 years ago

+1 from me.

There would need to be some sort of config to indicate which tests to attempt against each library. From my point of view I don't know if/when the sipsorcery library will even catch up to the list of features mentioned above let alone the full list of RFC's libwebrtc supports and of which more seem to regularly appear.

Sean-Der commented 3 years ago

100% agree

I can take a shot at ‘Docker and run’ next weekend. All yours if you want it earlier :)

Would you be interested in submitting to IETF hackathon? Don’t know if I can do live, but would get eyes on it!

sipsorcery commented 3 years ago

Yeah I'll take a look at an automated docker set up.

Would you be interested in submitting to IETF hackathon?

I'm guessing the benefit is exposure of a project to a wider audience?

sipsorcery commented 3 years ago

A GitHub Actions Workflow is now up and running to build docker images and run echo tests.

@Sean-Der I've added instructions for the Dockerfile. If you get a chance to test them out it would be nice to know if they are sufficient or not (they could well be missing steps I overlooked). Once you've got a Dockerfile add "pion" to the list of libraries in the workflow and the image should be automatically built.

jlaine commented 3 years ago

Do you think we could enable runs of the workflow for pull requests too? For instance, for my PR #19 which adds an aiortc client: I'd like to also include it as a client but don't want to discover I broke everything!

Obviously for PRs we don't want to have a change to the results commited :)

sipsorcery commented 3 years ago

Do you think we could enable runs of the workflow for pull requests too?

Sure done.

jlaine commented 3 years ago

Do you think we could enable runs of the workflow for pull requests too?

Sure done.

Hm, maybe this wasn't such a great idea after all. Currently the PR builds are failing, probably due to (legitimate) restrictions on access to the secrets.

Sean-Der commented 3 years ago

@sipsorcery sorry been busy :/

Submitting to the IETF hackathon might also get more acceptance of non-libwebrtc implementations! That is my big end goal. I would love to see a world where people build WebRTC projects with their favorite language.

Instead of saying 'Oh I use $x language for WebRTC support' it would be like building a HTTP/REST service. You use the language you are comfortable in :)