mod_http_upload not connecting to the put_url

[iPPLE]

I have followed your instruction but it is not working, on ejabberd log i see the request slot successfully but nothing uploaded to the destination http nginx server.

@mod_http_upload:create_slot/7:837 Got HTTP upload slot for 012345678@domain.com/d5a8f655-9758-4c73-a364-d1eb5db81ede (file: VOICE_896e8e90-01be-4f80-b966-835604bb7020.m4a, size: 14294)

Plz, help to advice. Thanks.

[licaon-kter]

nginx logs? error and access

gist of your configs? nginx and ejabberd

[iPPLE]

nginx logs? error and access

No logs created on nginx server.. it seems the upload request is not trigger

gist of your configs? nginx and ejabberd

• ejabberd.yml

  mod_http_upload:
      docroot: /var/www/html/upload
      put_url: http://upload.domain.com
      external_secret: "it-is-secret"

• /etc/nginx/site-available/upload.domain.com

server {
        listen 80;
        listen [::]:80;

        root /var/www/upload.domain.com/upload;
        index index.html index.htm index.nginx-debian.html;

        server_name upload www.upload.domain.com;

        location / {
                perl upload::handle;
                #try_files $uri $uri/ =404;
        }
        client_max_body_size 100m;
}

• /etc/nginx/nginx.conf

http {
        perl_modules /usr/local/lib/perl; # Path to upload.pm.
        perl_require upload.pm;
       ........
}

Note : ngx_http_perl_module.so already loaded by default, and i didn't change anything in /usr/local/lib/perl/upload.pm so the external_secret are the same.

nginx -t

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

[weiss]

Maybe the client (which one are you trying?) is insisting on using HTTPS?

[iPPLE]

Maybe the client (which one are you trying?) is insisting on using HTTPS?

I don't manage to use http(s) .. just for testing i think http should be fine... it has to be http(s) ?

Note : the slot is created successfully as i put the log there in ejabberd side:

* * * PutURL : <<"http://upload.domain.com/109f3eb1f575476ec9402cdd8501d48de3c697b5/YN8dcUs6ct4AQEBD89BEKOisVp2IjNJ9pCMuNXpG/VOICE_55bfd3eb-1064-411d-a524-83dae86e7cca.m4a">>
* * * GetURL : <<"http://upload.domain.com/109f3eb1f575476ec9402cdd8501d48de3c697b5/YN8dcUs6ct4AQEBD89BEKOisVp2IjNJ9pCMuNXpG/VOICE_55bfd3eb-1064-411d-a524-83dae86e7cca.m4a">>

And this function is not called in mod_http_upload , i'm using ejabberd 21.01.7

process(_LocalPath, #request{method = 'PUT', host = Host, ip = IP,
                 length = Length} = Request) ->

[weiss]

As I said the client might insist on HTTPS, some (most/all?) standard clients do that.

[iPPLE]

As I said the client might insist on HTTPS, some (most/all?) standard clients do that.

Yes, you are right.. after setup http(s) now .. i can see the upload request from ejabberd now but having one error:

2022/07/22 16:31:49 [error] 59471#59471: *3 perl: Rejecting upload: No auth token provided, client: 192.168.11.158, server: upload.domain.com, request: "PUT /109f3eb1f575476ec9402cdd8501d48de3c697b5/aecSXRWwGfmFba522nymaDsOEJ7EXGbXlHPiGiLs/VOICE_55bfd3eb-1064-411d-a524-83dae86e7cca.m4a HTTP/1.1", host: "upload.domain.com"

So what is wrong this time ? wrong external_secret ?

[weiss]

Rejecting upload: No auth token provided,

Your ejabberd.yml snippet included external_secret: "it-is-secret", but by staring at the code I really see no way to trigger this error message other than no external_secret being configured in the ejabberd.yml at all. Maybe you didn't restart the server since adding that, or something.

[iPPLE]

Rejecting upload: No auth token provided,

Your ejabberd.yml snippet included external_secret: "it-is-secret", but by staring at the code I really see no way to trigger this error message other than no external_secret being configured in the ejabberd.yml at all. Maybe you didn't restart the server since adding that, or something.

I debugged upload.pm, but it seems the condition is not matched $r->args =~ /v=([[:xdigit:]]{64})/,

sub handle_put {
    my $r = shift;
    my $len = $r->header_in('Content-Length') or return HTTP_LENGTH_REQUIRED;
    my $uri = $r->uri =~ s|(?:/[^/]+){$uri_prefix_components}/||r;
    my $provided_hmac;

    if (defined($r->args) and $r->args =~ /v=([[:xdigit:]]{64})/) {
        $provided_hmac = $1;
    } else {
        $r->log_error(0, 'Rejecting upload: No auth token provided');
        return HTTP_FORBIDDEN;
    }

    my $expected_hmac = hmac_sha256_hex("$uri $len", $external_secret);

    if (not safe_eq(lc($provided_hmac), lc($expected_hmac))) {
        $r->log_error(0, 'Rejecting upload: Invalid auth token');
        return HTTP_FORBIDDEN;
    }
    if (not $r->has_request_body(\&handle_put_body)) {
        $r->log_error(0, 'Rejecting upload: No data provided');
        return HTTP_BAD_REQUEST;
    }
    return OK;
}

And here is the log of $r->args:

Rejecting upload: No auth token provided, client: 192.168.11.158, server: upload.domain.com, request: "PUT /109f3eb1f575476ec9402cdd8501d48de3c697b5/AlKA0FvsT2aicNKvgMAjdbcmkCt2a6rdOEcbecEe/VOICE_242b8fac-59e0-4f3c-912c-a6b54588e235.m4a HTTP/1.1", host: "upload.domain.come"

It seems the auth token <<"?v=9decda3150c11bcc85b533843884a3d237f3c5158e40aa4fcb9d8abb98ea4648">> is not sent by ejabberd itself.

Note : i already restarted ejabberd everytime i changed ejabberd.yml.

[weiss]

It seems the auth token […] is not sent by ejabberd itself.

Yes, that's precisely what I meant. And I see no way this could happen except for mod_http_upload not seeing the external_secret option.

You could show me your entire ejabberd configuration (and the ejabberd version you're using) with any sensitive data stripped out, if you like.

[iPPLE]

It seems the auth token […] is not sent by ejabberd itself.

Yes, that's precisely what I meant. And I see no way this could happen except for mod_http_upload not seeing the external_secret option.

You could show me your entire ejabberd configuration (and the ejabberd version you're using) with any sensitive data stripped out, if you like.

I'm using ejabberd 21.01.7, i put the log on mod_http_upload and yes the external_secret is passed properly.

* * * external_secret : <<"0P5D1chY8SdLtg3QTQJZnhxwbeMBMQfvzRaatnor3QTYaaaasfweyteywqerwyrew">>

ejabberd.yml

###
###              ejabberd configuration file
###
### The parameters used in this configuration file are explained at
###
###       https://docs.ejabberd.im/admin/configuration
###
### The configuration file is written in YAML.
### *******************************************************
### *******           !!! WARNING !!!               *******
### *******     YAML IS INDENTATION SENSITIVE       *******
### ******* MAKE SURE YOU INDENT SECTIONS CORRECTLY *******
### *******************************************************
### Refer to http://en.wikipedia.org/wiki/YAML for the brief description.
###

include_config_file: ../../../etc/ejabberd/domain.yml
hosts:
- "dev.domain.com"
    oauth_expire: 157784760 # 5 years
    # OAuth token generation is enabled for all server users
    oauth_access: all
    # Check that the client ID is registered
    oauth_client_id_check: db

listen:
-
    port: 3478
    transport: udp
    module: ejabberd_stun
    use_turn: true
    turn_min_port: 49152
    turn_max_port: 65535
    ## The server's public IPv4 address:
    turn_ipv4_address: xxx.xx.xx.xx
-
    port: 5349
    transport: tcp
    module: ejabberd_stun
    use_turn: true
    tls: true
    turn_min_port: 49152
    turn_max_port: 65535
    ip: xxx.xx.xx.xx
    turn_ipv4_address: xxx.xx.xx.xx
-
    port: 5222
    ip: "::"
    module: ejabberd_c2s
    max_stanza_size: 262144
    shaper: c2s_shaper
    access: c2s
    starttls_required: true
-
    port: 5269
    ip: "::"
    module: ejabberd_s2s_in
    max_stanza_size: 524288
-
    port: 3443
    ip: "::"
    module: ejabberd_http
    tls: true
    request_handlers:
    /admin: ejabberd_web_admin
    /api: mod_http_api
    /bosh: mod_bosh
    /captcha: ejabberd_captcha
    /upload: mod_http_upload
    /ws: ejabberd_http_ws
-
    port: 5445
    ip: "::"
    module: ejabberd_http
    tls: true
    request_handlers:
    /oauth: ejabberd_oauth
-
    port: 5280
    ip: "::"
    module: ejabberd_http
    request_handlers:
    /bosh: mod_bosh
    /ws: ejabberd_http_ws
    /admin: ejabberd_web_admin
    /oauth: ejabberd_oauth
    /api: mod_http_api
    /.well-known/acme-challenge: ejabberd_acme
    web_admin: false
    http_bind: true
    # register: true
    #captcha: true
    tls: true

s2s_use_starttls: optional

acl:
    local:
        user_regexp: ""
    loopback:
        ip:
        - 127.0.0.0/8
        - ::1/128
    botuser:
        user:
        - "bot@dev.domain.com"

access_rules:

    local:
        allow: local
    c2s:
        deny: blocked
        allow: all
    announce:
        allow: admin
    configure:
        allow: admin
    muc_create:
        allow: local
    pubsub_createnode:
        allow: local
    trusted_network:
        allow: loopback

api_permissions:
    "bot api":
        who:
        access:
            allow:
            - acl: loopback
            - acl: botuser
        oauth:
            scope: ["get_roster","ejabberd:admin","sasl_auth"]
            access:
            allow: 
                - acl: botuser
        what:
        - "*"
        - "!stop"
        - "!start"

"console commands":
    from:
    - ejabberd_ctl
    who: all
    what: "*"
"admin access":
    who:
    access:
        allow:
        - acl: loopback
        - acl: admin
    oauth:
        scope: "ejabberd:admin"
        access:
        allow:
            - acl: loopback
            - acl: admin
    what:
    - "*"
    - "!stop"
    - "!start"
"public commands":
    who:
    ip: 127.0.0.1/8
    what:
    - status
    - connected_users_number

shaper:
    normal:
        rate: 3000
        burst_size: 20000
    fast: 100000

shaper_rules:
    max_user_sessions: 10
    max_user_offline_messages:
        5000: admin
        100: all
    c2s_shaper:
        none: admin
        normal: all
    s2s_shaper: fast

domain.yml

default_db: sql
sm_db_type: sql
auth_password_format: scram
auth_use_cache: false
# loglevel: Verbosity of log files generated by ejabberd.
## 0: No ejabberd log at all (not recommended)
## 1: Critical
## 2: Error
## 3: Warning
## 4: Info
## 5: Debug
loglevel: 4
imodiqdisc: 100
released: 0
appname:  "domain"
certfiles:
    - "/usr/local/trendsec/domain_ssl/domain.pem"

## By default the frequency of account registrations from the same IP
## is limited to 1 account every 10 minutes. To disable, specify: infinity
## registration_timeout: 600
registration_timeout: 1

host_config:
    "dev.domain.com":
        sql_type: pgsql
        sql_server: "localhost"
        sql_database: "chat"
        sql_username: "postgres"
        sql_password: "password"
        sql_port: 5432
        auth_method: [sql]

append_host_config:
    "dev.domain.com":
        modules:
            mod_admin_update_sql: {}
            mod_adhoc: {}
            mod_admin_extra: {}
            mod_announce:
                access: announce
            mod_avatar: {}
            mod_blocking: {}
            mod_bosh: 
                ram_db_type: sql
                cache_life_time: 3600 ## 1 hour
            mod_caps: {}
            mod_carboncopy: {}
            mod_client_state: {}
            mod_configure: {}
            mod_disco: {}
            # mod_fail2ban: {}
            mod_http_api: {}
            mod_last: {}
            mod_mam:
                ## Mnesia is limited to 2GB, better to use an SQL backend
                ## For small servers SQLite is a good fit and is very easy
                ## to configure. Uncomment this when you have SQL configured:
                ## db_type: sql
                user_mucsub_from_muc_archive: true
                assume_mam_usage: true
                default: always
            mod_mqtt: {}
            mod_offline:
                access_max_user_messages: max_user_offline_messages
            mod_ping: {}
            mod_privacy: {}
            mod_private:
                use_cache: false
            mod_register:
                ## Only accept registration requests from the "trusted"
                ## network (see access_rules section above).
                ## Think twice before enabling registration from any
                ## address. See the Jabber SPAM Manifesto for details:
                ## https://github.com/ge0rg/jabber-spam-fighting-manifesto
                # ip_access: trusted_network
                ip_access: all
            mod_roster:
                use_cache: false
                versioning: true
            mod_s2s_dialback: {}
            mod_shared_roster: {}
            mod_stream_mgmt:
                max_resume_timeout: 1
                resume_timeout: 1
                resend_on_timeout: false
                # resend_on_timeout: if_offline
            mod_stun_disco:
                secret: secret
                credentials_lifetime: 12h
                services:
                    -
                    host: xxx.xx.xx.x
                    port: 3478
                    type: stun
                    transport: udp
                    restricted: false
                    -
                    host: xxx.xx.xx.x
                    port: 3478
                    type: turn
                    transport: udp
                    restricted: true
                    -
                    host: dev.domain.com
                    port: 5349
                    type: stuns
                    transport: tcp
                    restricted: false
                    -
                    host: dev.domain.com
                    port: 5349
                    type: turns
                    transport: tcp
                    restricted: true
            mod_time: {}
            mod_vcard: {}
            mod_vcard_xupdate: {}
            mod_version:
                show_os: false
            mod_http_upload:
                put_url: https://upload.domain.com
                external_secret: "0P5D1chY8SdLtg3QTQJZnhxwbeMBMQfvzRaatnor3QTYaaaasfweyteywqerwyrew"
                thumbnail: true
                max_size: 1048576000
            mod_pubsub:
                access_createnode: pubsub_createnode
                plugins:
                - flat
                - pep
                force_node_config:
                ## Avoid buggy clients to make their bookmarks public
                storage:bookmarks:
                    access_model: whitelist

            mod_mix: {}
            mod_muc:
                access:
                - allow
                access_admin:
                - allow: admin
                access_create: muc_create
                access_persistent: muc_create
                access_mam:
                - allow
                default_room_options:
                allow_subscription: true
                allow_change_subj: true
                allow_private_messages: true
                allow_query_users: true
                allow_user_invites: true
                allow_visitor_nickchange: true
                anonymous: false
                members_by_default: true
                members_only: true
                public: false
                mam: true
                max_users: 200
                persistent: false
                #mod_muc_log: {}
            mod_muc_admin: {}

Thanks.

[iPPLE]

It seems the auth token […] is not sent by ejabberd itself.

Yes, that's precisely what I meant. And I see no way this could happen except for mod_http_upload not seeing the external_secret option.

You could show me your entire ejabberd configuration (and the ejabberd version you're using) with any sensitive data stripped out, if you like.

Sorry it was my fault to modify mod_http_upload by excluding Query form the PutURL1 PutURL1 = <<(misc:url_encode(PutURL))/binary, Query/binary>>, and now I can see the Auth token = %3Fv%3Dc1d65f130f65137ae2e40931d333a25583b3f684247524ea38899e0fb351ea8c passed to nginx server now but still it said:

2022/07/26 14:11:13 [error] 2151#2151: *157 perl: Rejecting upload: No auth token provided, client: 192.168.11.158, server: upload.domain.com, request: "PUT /109f3eb1f575476ec9402cdd8501d48de3c697b5/c7E3wr4KgxxzFHmtHFDADHzWKvzdvwnpOqrqeum7/VOICE_e3927379-1cdb-410a-ae8c-f7875f8c7985.m4a%3Fv%3Dc1d65f130f65137ae2e40931d333a25583b3f684247524ea38899e0fb351ea8c HTTP/1.1", host: "upload.domain.com"

[iPPLE]

The URL is encoded v%3D, it should be v= so that the condition is matched in upload.pm, or is there something missing in my nginx configuration ? Plz help to advice... i'm new to perl or nginx.

[weiss]

The URL is encoded v%3D

Maybe the client is doing that blindly? Which one are you using?

[iPPLE]

The URL is encoded v%3D

Maybe the client is doing that blindly? Which one are you using?

Ah i see.. i will check with mobile developer.. thanks.

[weiss]

i will check with mobile developer

URLs are returned properly encoded by the server (as per the spec), so the client can just use them as-is.

I'll close this issue for the moment. If something doesn't work using the unmodified ejabberd code with one of the (freely available) standard XMPP clients, just reopen a new issue.

[iPPLE]

i will check with mobile developer

URLs are returned properly encoded by the server (as per the spec), so the client can just use them as-is.

I'll close this issue for the moment. If something doesn't work using the unmodified ejabberd code with one of the (freely available) standard XMPP clients, just reopen a new issue.

Thanks for your kindness support. @weiss