search

harvard-dce / zoom-recording-ingester

A set of AWS services for downloading and ingesting Zoom meeting videos into Opencast
Apache License 2.0
23 stars 5 forks source link

Can't deploy project zoom-recording-ingester #119

Open newgate1999 opened 3 years ago

newgate1999 commented 3 years ago

Dear all, I'm try to deploy project to my server but it's not working. The log server:

uploading schedule-update to s3://schoolonair-bbb/zoom-upload/schedule-update.zip est: Stacks[?Name=='zoom-upload'].VpcId stacks: {'Stacks': [], 'ResponseMetadata': {'RequestId': 'dedb0838-7e31-4e33-bd28-49fb15d4966d', 'HTTPStatusCode': 200, 'HTTPHeaders': {'content-type': 'application/x-amz-json-1.1', 'date': 'Thu, 04 Mar 2021 12:59:35 GMT', 'x-amzn-requestid': 'dedb0838-7e31-4e33-bd28-49fb15d4966d', 'content-length': '13', 'connection': 'keep-alive'}, 'RetryAttempts': 0}} Traceback (most recent call last): File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/newgate/Documents/aws/zoom-recording-ingester/cdk/app.py", line 28, in oc_vpc_id, oc_security_group_id = vpc_components() File "/home/newgate/Documents/aws/zoom-recording-ingester/cdk/helpers.py", line 31, in vpc_components vpc_id = jmespath.search( IndexError: list index out of range Subprocess exited with error 1

In readme file no detailed instructions on config file env. If possible, can't you give me a video tutorial or more detailed config steps ?

Regard,

lbjay commented 3 years ago

Hi,

First off, a disclaimer that we put this project's code on github with a license but don't support or develop it outside of DCE's needs.

The settings in the example .env file have some comments explaining a bit about their use/purpose.

I'm not sure what you mean by deploying the project to your server. Did you try running invoke stack.create?

newgate1999 commented 3 years ago

I'm try to running invoke stack.create. But it looks like the stacks have not been created and response emply array

nataliemona commented 3 years ago

The error you posted:

File "/home/newgate/Documents/aws/zoom-recording-ingester/cdk/helpers.py", line 31, in vpc_components
vpc_id = jmespath.search(
IndexError: list index out of range
Subprocess exited with error 1

Shows that you are missing a VPC for your Opencast cluster. This isn't mentioned in prerequisites but the application does expect that the Opencast cluster has a VPC setup.

nataliemona commented 3 years ago

@newgate1999 Were you able to get it to work?

newgate1999 commented 3 years ago

I tried but it doesn't work. Can you shoot a video setup and deploy code for me to refer ? Thank you so much.

newgate1999 commented 3 years ago

This is my config. can you check it for me ?

# should only be necessary if you have multiple credential profiles
AWS_PROFILE=phamtuananh

# tags that will be assigned to the cfn stack and all resources under it
# space-separated key/value pairs, like so:
# Key=key1,Value=value1 Key=key2,Value=value2
STACK_TAGS=Key=project,Value=MH Key=department,Value=DE

# name of the cfn stack; also used to prefix many resource names (required)
STACK_NAME=zoom-upload

# s3 bucket to store packaged lambda code
LAMBDA_CODE_BUCKET=schoolonair-bbb

# where cloudwatch alarm notifications will get sent
NOTIFICATION_EMAIL=tuananh301099@gmail.com

# Email address associated with a Zoom administrative level account
# This is used to get a token to authenticate downloads
ZOOM_ADMIN_EMAIL=share.schoolonair2022@gmail.com

ZOOM_API_BASE_URL=
# Either a zoom api key/secret or an apigee key. If both are present then
# apigee will be prefered.
ZOOM_API_KEY=*****************
ZOOM_API_SECRET=************************
APIGEE_KEY=

# URL and API auth for the target opencast cluster
OC_CLUSTER_NAME=test-s3
OPENCAST_API_USER=admin
OPENCAST_API_PASSWORD=opencast

# id of series to receive ingests in the absence of a zoom meeting -> series mapping
# this is meant for testing/dev only. If you set this to a valid Opencast series id
# then any recording that doesn't match something in the class schedule data will
# be ingested into that default series id.
DEFAULT_SERIES_ID=

# Opencast 5.x:
#     publisher = producer username or email
#     no contributor
# When the publisher of a series cannot be determined via an
# /otherpubs/epidodedefault lookup, the Opencast workflow notifications will go
# to this address. If this is left empty the notifications will go to the
# NOTIFICATION_EMAIL address.
DEFAULT_PUBLISHER=admin
# These settings will override the Opencast lookup of publisher and contributor
# for a series.
OVERRIDE_PUBLISHER=admin
OVERRIDE_CONTRIBUTOR=admin

# 5x clusters: "dce-int-production-zoom" / "multipart/chunked+source"
OC_WORKFLOW=dce-int-production-zoom
OC_FLAVOR=multipart/chunked+source

# controls how many download queue messages will be processed (not ingested!) per
# invocation of the downloader function
DOWNLOAD_MESSAGES_PER_INVOCATION=10

# controls how far in minutes the schedule matching will allow for start/end times
BUFFER_MINUTES=30

# videos shorter than this many minutes will be ignored
MINIMUM_DURATION=2

# Python pytz timezone
LOCAL_TIME_ZONE=US/Eastern

LOG_NOTIFICATIONS_FILTER_LOG_LEVEL=ERROR

# Password for the opencast mysql root user
# You can get this from the cluster config of the cluster being ingested to
# The invoke tasks will find the actual RDS endpoint via an aws lookup
OC_DB_PASSWORD=opencast

# the uploader will query opencast for the number of currently running track uploads
# if it is greater than this number the uploader will abort (leaving the upload in the queue)
OC_TRACK_UPLOAD_MAX=5

# These (comma-separated) IPs will be included in the APIs resource policy
# Only requests coming from these IPs will be allowed to exec the on-demand ingest endpoint
# Assuming the Opsworks cluster is up-to-date, these should be the same ips listed
# in the cluster config's "vpn_ips" list.
INGEST_ALLOWED_IPS=

# Google sheets
# The document id
GSHEETS_DOC_ID =
# The name of the actual tab to download as a csv
GSHEETS_SHEET_NAME=
nataliemona commented 3 years ago

It looks like you might be missing the ZOOM_API_BASE_URL= value, the gsheets values, and the ingest allowed ips? I don't currently have time to record a tutorial video, but I'm happy to help point you in the right direction if you can show me a specific error.

newgate1999 commented 3 years ago

Thanks nataliemona. I am having an error in the opencast server cluster. I don't know how to install and pass opencast related parameters to .env file. OC_CLUSTER_NAME=?, OC_WORKFLOW = ?, OC_FLAVOR = ?. My cluster name does not exist. It looks like besides installing opencast I have to install project https://github.com/harvard-dce/mh-opsworks ,right?

nataliemona commented 3 years ago

It might make more sense for you to fork the project and then adapt it to work with whatever Opencast setup you have.

Srinivasysr2003 commented 3 years ago

I am getting error as below and I have private subnet in which opencast has created. jsii.errors.JSIIError: Not possible to place Lambda Functions in a Public subnet Subprocess exited with error 1

function def vpc_components(): trying to consider only VPC ID and Security Group. Where is mentioned to provided subnet group id?