Help adding this to my Lambda Function.

[wes1993]

Hello, i'm trying to add this to my AlexaLambda function instead of using pytube or youtube_dl but i can't understand how this works? this pass a Json response or something else?

How have you added this to my Lambda function?

Best Regards Stefano

[DavidBerdik]

Hello!

This application effectively acts as a proxy between the client and YouTube. It's like what the RapidAPI support you added used to do. This application uses yt-dlp (a fork of youtube-dl that has better community support and is more actively maintained) to process the YouTube stream and repeat it back to the client in a much more friendly format, so, instead of returning a YouTube stream URL directly, the Lambda function would return a URL that points to this application, and when Alexa accesses the URL, the application handles everything. I have been using it with my setup for many months now, and it works very well. The only issue is that you need to host this application on a server that Alexa can access. In my case, since I already have a NAS server, I just run it in a Docker container on that server.

The general flow of using this in a Lambda function would be the following:

1. Use the YouTube API to retrieve the list of video IDs.
2. Return a stream URL of the following format to Alexa: https://<IP address or hostname>:<PORT NUMBER>/api/dl/<VIDEO ID>?f=bestaudio
3. Alexa will connect to the returned URL, and the application will begin serving an audio stream of the given video ID to Alexa.

If you want to see the implementation that I use, take a look at this commit.

There are a few things to keep in mind when looking at my implementation:

• I do not have any Alexa devices with displays, so I have not implemented support for playback with video. The f parameter passed through the URL controls what is passed to the f flag for yt-dlp, so you might be able to get it to work by setting bestvideo. I do not know for sure though, as I have not (and cannot) test it.
• Alexa audio streams are required to be served over HTTPS, and this application only serves requests over HTTP. To work around this, I use a Dockerized reverse proxy server (this one, to be specific) configured with Let's Encrypt. I configured the reverse proxy server to accept HTTPS requests, and forward them as HTTP requests to the Docker container for the YouTube Stream Repeater.
• You probably do not want to expose a running instance of YouTube Stream Repeater to the internet, as anyone who finds your server could then use it, which can obviously be quite resource-intensive. Unfortunately, for it to work with Alexa, you have no choice. Since I do not want any random client to be able to use my server, I generated a preshared key, which I then configured the reverse proxy server to check the URL's GET parameters for. When my Lambda function returns a URL to Alexa, it includes this preshared key in the URL. If the key is not present, then the reverse proxy server drops the request. This is definitely not a real security solution. There are a ton of things wrong with this approach, but for my needs, it is sufficient. The code for the Lambda function component of this preshared key mechanism is available in this commit.

[wes1993]

Thanks a lot @DavidBerdik for your reply,

I i'm trying to make your same configuration

Thanks a lot Stefano.

[DavidBerdik]

@wes1993 You're welcome! Let me know if you have any other questions!

[wes1993]

Finally i have created an HomeAssistant addon related to this, so for someone that use HA can use the addon instead of install directly in Docker.

Thanks seems that works really well and faster/better than pytube/youtube_dl.

When we have time we could implement an HTTPS server and Username/Password authentication (Basic HTTP) so we can use just this instead of using a different proxy ;-D

[DavidBerdik]

The Home Assistant addon is a great idea! I am not a Home Assistant user, so this thought had not even occurred to me. Did you make the addon public?

You're welcome! From my experience with using it so far, it breaks on occasion, but it is overall much more stable.

If you want to run the server without relying on a reverse proxy, the instructions for configuring Uvicorn (the web server that this application uses) with HTTPS are available here. For my use case, I have multiple internet-facing applications that need HTTPS, so it makes more sense to use a single reverse proxy to handle HTTPS rather than setting up individual configurations for each one. As for implementing username and password authentication, I am pretty sure that this is impossible to use because Alexa does not support it.

[wes1993]

Hello Dave, sure, this is the repo with your YouTube-Stream-Repeater converted in addon: https://github.com/wes1993/Wes93-Repo

Regarding the username and password i can confirm that works well, i have configured my proxy with basic authentication and in the Lambda i have used this URL format:

http://<User>:<Pass>@<IP>:<Port>/

The bad thing is that seems that Uvicorn won't support Basic HTTP authentication.. :-(

Bye Stefano

[DavidBerdik]

Hello Stefano,

That's great! Thank you for sharing it.

That's interesting! I thought I had read somewhere that Alexa doesn't support it for audio streams.

As for supporting Basic HTTP Auth, I'm pretty sure you can do it. Take a look at this documentation.

~ David

[wes1993]

Thanks a lot David for your link!!!

I will try to make a lot :-D

The problem is that i don't know how to pass the variable username and password from Homeassistant config addon to rapidapi

Bye Stefano

[wes1993]

Do you have discord or telegram?

[wes1993]

Hello, finally i have configured the container also with https, the last this is that i can't make the basic authentication to work...

This is the code that i have tried to adapt but unfrotunately won't work and return null (Seems that can't handle the video_id):

import mimetypes
import os

import magic
import secrets

from fastapi import Depends, FastAPI, HTTPException, status
from fastapi.responses import StreamingResponse, JSONResponse, FileResponse
from fastapi.background import BackgroundTasks
from fastapi.security import HTTPBasic, HTTPBasicCredentials

from util.sub import download_subs
from util.meta import query_meta
from util.stream import stream_from_yt

app = FastAPI()

security = HTTPBasic()

def get_current_username(credentials: HTTPBasicCredentials = Depends(security)):
    current_username_bytes = credentials.username.encode("utf8")
    correct_username_bytes = b"stanleyjobson"
    is_correct_username = secrets.compare_digest(
        current_username_bytes, correct_username_bytes
    )
    current_password_bytes = credentials.password.encode("utf8")
    correct_password_bytes = b"swordfish"
    is_correct_password = secrets.compare_digest(
        current_password_bytes, correct_password_bytes
    )
    if not (is_correct_username and is_correct_password):
        raise HTTPException(
            status_code=status.HTTP_401_UNAUTHORIZED,
            detail="Incorrect email or password",
            headers={"WWW-Authenticate": "Basic"},
        )
    return credentials.username

@app.get("/")
def read_current_user(username: str = Depends(get_current_username)):
    return {"usefghfghfhrname": username}

@app.get("/dl/{video_id}")
def read_current_user(username: str = Depends(get_current_username)):
#    return {"username": username}
    async def api_dl(
        video_id: str,   # the video's ID (watch?v=<this>)
        f: str = "best", # format 
        sl: str = None,  # subtitle language to embed
    ):
        stream = stream_from_yt(video_id, f, sl)  
        first_chunk = await stream.__anext__() # peek first chunk

        # guess filetype
        m = magic.Magic(mime=True, uncompress=True)
        mime_type = m.from_buffer(first_chunk)

        # guess extension based on mimetype
        ext = mimetypes.guess_extension(mime_type) or '.mkv' # fallback to mkv I guess

        print(f"[{video_id}]: download type: {mime_type} ({ext})")

        headers = {
            "Content-Disposition": f"attachment;filename={video_id}{ext}"
        }

        async def joined_stream():
            # attach the first chunk back to the generator
            yield first_chunk

            # pass on the rest
            async for chunk in stream:
                yield chunk

        # pass that to the user
        return StreamingResponse(
            joined_stream(),
            media_type = mime_type,
            headers = headers
        )

@app.get("/meta/{video_id}")
def read_current_user(username: str = Depends(get_current_username)):
#    return {"username": username}
    async def api_meta(video_id: str):
        meta = query_meta(video_id)

        if meta is None:
            raise HTTPException(
                status_code=400, 
                detail="Could not get meta for requested Video ID!"
            )

        return JSONResponse(meta)

    def _remove_file(path: str) -> None:
        if not (
            path.endswith('.vtt')
            or path.endswith('.srt')
            or path.endswith('.ass')
        ):
            # don't delete weird files
            # better safe than sorry
            return
        os.remove(path)

@app.get("/sub/{video_id}")
def read_current_user(username: str = Depends(get_current_username)):
#    return {"username": username}
    async def api_sub(background_tasks: BackgroundTasks, video_id: str, l: str = "en", f: str = "vtt"):
        if f not in ["vtt", "ass", "srt"] and not (l == "live_chat" and f == "json"):
            raise HTTPException(
                status_code=400,
                detail="Invalid subtitle format, valid options are: vtt, ass, srt"
            )

        sub_file = download_subs(video_id, l, f)

        background_tasks.add_task(_remove_file, sub_file)

        return FileResponse(
            sub_file,
            filename=f"{video_id}.{l}.{f}"
        )

Do you have some suggestion?

Best regards Stefano

[DavidBerdik]

Thanks a lot David for your link!!!

I will try to make a lot :-D

The problem is that i don't know how to pass the variable username and password from Homeassistant config addon to rapidapi

Bye Stefano

You're welcome! I assume that instead of Rapid API, you meant you can't figure out how to do it with this YouTube Stream Repeater? Either way, I unfortunately can't help, as I do not have any experience with developing for Home Assistant.

Do you have discord or telegram?

I do! If you want to connect with me, send an e-mail to the address listed on my GitHub profile, and I'll share it with you privately.

Do you have some suggestion?

So, I haven't tested this, and therefore have no idea if it works, but I believe that what you want is something like this:

@app.get("/dl/{video_id}")
async def api_dl(
    video_id: str,   # the video's ID (watch?v=<this>)
    f: str = "best", # format 
    sl: str = None,  # subtitle language to embed
    username: str = Depends(get_current_username)
):
    stream = stream_from_yt(video_id, f, sl)  
    first_chunk = await stream.__anext__() # peek first chunk

    # guess filetype
    m = magic.Magic(mime=True, uncompress=True)
    mime_type = m.from_buffer(first_chunk)

    # guess extension based on mimetype
    ext = mimetypes.guess_extension(mime_type) or '.mkv' # fallback to mkv I guess

    print(f"[{video_id}]: download type: {mime_type} ({ext})")

    headers = {
        "Content-Disposition": f"attachment;filename={video_id}{ext}"
    }

This is just a slightly modified version of this code. Actually, the only thing I added is the username: str = Depends(get_current_username) parameter. From what I understand, the Depends() thing will call get_current_username(), which contains the code for validating the credentials that were provided. If the credentials are valid, then api_dl will execute, and if the credentials are invalid, then get_current_username() will raise an HTTPException, which will cause the attempted request to be rejected.

[wes1993]

Hello @DavidBerdik Thanks again a lot for your reply :-D I have sent you an email :-D

I have added your code and works well for the /dl/ but how can i add in the other two function (meta, sub)?

This is the code that i have tested but won't work :-(

@app.get("/meta/{video_id}")
#def read_current_user(username: str = Depends(get_current_username)):
#    return {"username": username}
async def api_meta(video_id: str):
    username: str = Depends(get_current_username)
    meta = query_meta(video_id)
    if meta is None:
        raise HTTPException(
            status_code=400, 
            detail="Could not get meta for requested Video ID!"
        )

    return JSONResponse(meta)

def _remove_file(path: str) -> None:
    if not (
        path.endswith('.vtt')
        or path.endswith('.srt')
        or path.endswith('.ass')
    ):
        # don't delete weird files
        # better safe than sorry
        return
    os.remove(path)

@app.get("/sub/{video_id}")
#def read_current_user(username: str = Depends(get_current_username)):
#    return {"username": username}
async def api_sub(background_tasks: BackgroundTasks, video_id: str, l: str = "en", f: str = "vtt"):
    username: str = Depends(get_current_username)
    if f not in ["vtt", "ass", "srt"] and not (l == "live_chat" and f == "json"):
        raise HTTPException(
            status_code=400,
            detail="Invalid subtitle format, valid options are: vtt, ass, srt"
        )

    sub_file = download_subs(video_id, l, f)

    background_tasks.add_task(_remove_file, sub_file)

    return FileResponse(
        sub_file,
        filename=f"{video_id}.{l}.{f}"
    )

[DavidBerdik]

Hello @wes1993,

You're welcome! I think to make the authentication work with the other functions, you can just take the existing code from my repo and add the username: str = Depends(get_current_username) parameter to them. I don't think any other changes should be necessary.

~ David