search

laravel / octane

Supercharge your Laravel application's performance.
https://laravel.com/docs/octane
MIT License
3.78k stars 296 forks source link

Stream File Download with Octane/Swoole gives blank file #260

Closed dbelyaeff closed 3 years ago

dbelyaeff commented 3 years ago

Description:

Running file downloading from remote server gives blank file (0kb) when octane is enabled.

public function download(Video $video)
    {
        return response()->streamDownload(function () use ($video) {
            echo file_get_contents($video->mp3);
        }, basename($video->mp3));
    }

Switching back to sail artisan server works fine – full length file is downloaded.

Steps To Reproduce:

  1. Install Laravel.
  2. Install Sail.
  3. Install Octane for Sail via Octane github docs.
  4. Create simple route with callback like the given one upwards.
  5. Try to stream any remote file (img, whatever).
  6. Blank file is downloaded (0 bytes).
  7. Switch back from octane to standard Sail/Laravel artisan server.
  8. Now remote file is downloaded via stream correctly, w/full size.
nunomaduro commented 3 years ago

I don't have Sail currently installed - yet, without sail seems to work as expected when simply using octane:start with both RoadRunner and Swoole. Is someone else able to reproduce this issue?

sy-records commented 3 years ago

What swoole configuration items have you set

dbelyaeff commented 3 years ago

I've basically followed the instructions provided via Octanes github README.md. So here's a part of supervisord.conf:

command=/usr/bin/php -d variables_order=EGPCS /var/www/html/artisan octane:start --watch --server=swoole --host=0.0.0.0 --port=80
nunomaduro commented 3 years ago

I am sorry - but I was unable to reproduce your issue, even after following the "Steps To Reproduce" in your issue description.

If other developers are facing the same issue please let me know, meanwhile I believe this must be something related to your environment/setup.

fridzema commented 3 years ago

@nunomaduro I am currently facing the same issue altough a little bit different. 

laravel/framework v8.42.0 
laravel/octane v1.0.1
php 8.0.5
Ubuntu 20.04.2 (forge provisioned)

php --ri swoole

Swoole => enabled
Author => Swoole Team <team@swoole.com>
Version => 4.6.5
Built => Apr 13 2021 11:47:21
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
pcre => enabled
zlib => 1.2.11
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 8388608 => 8388608

I am streaming basic pdf files that come from a third party api. Assets around ~80mb gives me a 0kb "corrupt" file, where the smaller files download just as expected. 

    return response()->streamDownload(function () use ($file_data) {
      echo base64_decode($file_data['data']);
    }, $file_data['file_name']);

First things i tried myself before i found this issue, increasing php_memory, disabled opcache etc. with no succes. When i was looking in the logs i stumbled upon the following interesting values package_max_length and socket_buffer_size :

{
    "masterProcessId": 1654,
    "managerProcessId": 1733,
    "state": {
        "appName": "DEV",
        "host": "127.0.0.1",
        "port": "8000",
        "workers": 2,
        "taskWorkers": 2,
        "maxRequests": "500",
        "publicPath": "\/home\/forge\/DOMAIN\/public",
        "storagePath": "\/home\/forge\/DOMAIN\/storage",
        "defaultServerOptions": {
            "buffer_output_size": 10485760,
            "enable_coroutine": false,
            "daemonize": false,
            "log_file": "\/home\/forge\/DOMAIN\/storage\/logs\/swoole_http.log",
            "log_level": 5,
            "max_request": "500",
            "package_max_length": 10485760,
            "reactor_num": 2,
            "send_yield": true,
            "socket_buffer_size": 10485760,
            "task_max_request": "500",
            "task_worker_num": 2,
            "worker_num": 2
        },
        "octaneConfig": {
            "server": "swoole",
            "https": true,
            "listeners": {
                "Laravel\\Octane\\Events\\WorkerStarting": [
                    "Laravel\\Octane\\Listeners\\EnsureUploadedFilesAreValid"
                ],
                "Laravel\\Octane\\Events\\RequestReceived": [
                    "Laravel\\Octane\\Listeners\\CreateConfigurationSandbox",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToAuthorizationGate",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToBroadcastManager",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToDatabaseManager",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToHttpKernel",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToMailManager",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToNotificationChannelManager",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToPipelineHub",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToQueueManager",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToRouter",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToValidationFactory",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToViewFactory",
                    "Laravel\\Octane\\Listeners\\FlushDatabaseRecordModificationState",
                    "Laravel\\Octane\\Listeners\\FlushArrayCache",
                    "Laravel\\Octane\\Listeners\\PrepareInertiaForNextOperation",
                    "Laravel\\Octane\\Listeners\\PrepareScoutForNextOperation",
                    "Laravel\\Octane\\Listeners\\PrepareSocialiteForNextOperation",
                    "Laravel\\Octane\\Listeners\\FlushLocaleState",
                    "Laravel\\Octane\\Listeners\\FlushQueuedCookies",
                    "Laravel\\Octane\\Listeners\\FlushSessionState",
                    "Laravel\\Octane\\Listeners\\FlushAuthenticationState",
                    "Laravel\\Octane\\Listeners\\EnforceRequestScheme",
                    "Laravel\\Octane\\Listeners\\EnsureRequestServerPortMatchesScheme",
                    "Laravel\\Octane\\Listeners\\GiveNewRequestInstanceToApplication"
                ],
                "Laravel\\Octane\\Events\\RequestHandled": [],
                "Laravel\\Octane\\Events\\RequestTerminated": [],
                "Laravel\\Octane\\Events\\TaskReceived": [
                    "Laravel\\Octane\\Listeners\\CreateConfigurationSandbox",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToAuthorizationGate",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToBroadcastManager",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToDatabaseManager",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToHttpKernel",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToMailManager",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToNotificationChannelManager",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToPipelineHub",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToQueueManager",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToRouter",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToValidationFactory",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToViewFactory",
                    "Laravel\\Octane\\Listeners\\FlushDatabaseRecordModificationState",
                    "Laravel\\Octane\\Listeners\\FlushArrayCache",
                    "Laravel\\Octane\\Listeners\\PrepareInertiaForNextOperation",
                    "Laravel\\Octane\\Listeners\\PrepareScoutForNextOperation",
                    "Laravel\\Octane\\Listeners\\PrepareSocialiteForNextOperation"
                ],
                "Laravel\\Octane\\Events\\TickReceived": [
                    "Laravel\\Octane\\Listeners\\CreateConfigurationSandbox",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToAuthorizationGate",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToBroadcastManager",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToDatabaseManager",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToHttpKernel",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToMailManager",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToNotificationChannelManager",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToPipelineHub",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToQueueManager",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToRouter",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToValidationFactory",
                    "Laravel\\Octane\\Listeners\\GiveNewApplicationInstanceToViewFactory",
                    "Laravel\\Octane\\Listeners\\FlushDatabaseRecordModificationState",
                    "Laravel\\Octane\\Listeners\\FlushArrayCache",
                    "Laravel\\Octane\\Listeners\\PrepareInertiaForNextOperation",
                    "Laravel\\Octane\\Listeners\\PrepareScoutForNextOperation",
                    "Laravel\\Octane\\Listeners\\PrepareSocialiteForNextOperation"
                ],
                "Laravel\\Octane\\Contracts\\OperationTerminated": [
                    "Laravel\\Octane\\Listeners\\FlushTemporaryContainerInstances",
                    "Laravel\\Octane\\Listeners\\DisconnectFromDatabases",
                    "Laravel\\Octane\\Listeners\\CollectGarbage"
                ],
                "Laravel\\Octane\\Events\\WorkerErrorOccurred": [
                    "Laravel\\Octane\\Listeners\\ReportException",
                    "Laravel\\Octane\\Listeners\\StopWorkerIfNecessary"
                ],
                "Laravel\\Octane\\Events\\WorkerStopping": []
            },
            "warm": [
                "auth",
                "cache",
                "cache.store",
                "config",
                "cookie",
                "db",
                "db.factory",
                "encrypter",
                "files",
                "hash",
                "log",
                "router",
                "routes",
                "session",
                "session.store",
                "translator",
                "url",
                "view"
            ],
            "flush": [],
            "cache": {
                "rows": 1000,
                "bytes": 10000
            },
            "tables": {
                "example:1000": {
                    "name": "string:1000",
                    "votes": "int"
                }
            },
            "watch": [
                "app",
                "bootstrap",
                "config",
                "database",
                "public\/**\/*.php",
                "resources\/**\/*.php",
                "routes",
                "composer.lock",
                ".env"
            ],
            "garbage": 50,
            "max_execution_time": 10
        }
    }

Is there anybody who know how to increase the package_max_length and socket_buffer_size as i could only find it "hardcoded" in StartSwooleCommand.php or am i looking in the wrong direction?

sy-records commented 3 years ago

@fridzema

config/octane.php

    'swoole' => [
        'options' => [
        ],
    ],
fridzema commented 3 years ago

@sy-records Looks like it is a step in the right direction!

Had to increase the values to a massive ammount:

  'swoole' => [
    'options' => [
      'package_max_length' => 100000000,
      'socket_buffer_size' => 100000000,
      'buffer_output_size' => 100000000,
    ],
  ],

The weird thing i am facing right now is that sometimes it works, and sometimes i get after around ~30mb suddenly a network error. Very strange...

Currently trying to reproduce it an seperate repo to share here.

fridzema commented 3 years ago

i have setup a repo that demonstrates the problem: https://github.com/fridzema/stream-download-octane-sample

I also made a short clip to demonstrate: https://user-images.githubusercontent.com/8180660/119002043-ce24a480-b98c-11eb-8689-56de50f0fa11.mp4

The sample files i used for testing: https://github.com/fridzema/stream-download-octane-sample/releases/download/0.0.1/sample-files.zip

sy-records commented 3 years ago

@fridzema The problem was reproduced and this option was removed to fix it.

286 After the merge, you can upgrade Swoole to v4.6.7 without setting 'buffer_output_size' => 100000000,