Don't setup Scylla in developer mode in `scyllamgr_setup` script

[Michal-Leszczynski]

Ref https://github.com/scylladb/scylla-enterprise/issues/3867#issuecomment-1926639568. Note that SM does not even run scylla_setup script.

[Michal-Leszczynski]

From what I can see in Scylla's main.cc:

adjust_and_verify_rlimit(cfg->developer_mode());
verify_adequate_memory_per_shard(cfg->developer_mode());
verify_seastar_io_scheduler(opts, cfg->developer_mode());
...
...
...
verify_adequate_memory_per_shard(bool developer_mode) {
    auto shard_mem = memory::stats().total_memory();
    if (shard_mem >= (1 << 30)) {
        return;
    }
    if (developer_mode) {
        startlog.warn("Only {} MiB per shard; this is below the recommended minimum of 1 GiB/shard;"
                " continuing since running in developer mode", shard_mem >> 20);
    } else {
        startlog.error("Only {} MiB per shard; this is below the recommended minimum of 1 GiB/shard; terminating."
                "Configure more memory (--memory option) or decrease shard count (--smp option).", shard_mem >> 20);
        throw std::runtime_error("configuration (memory per shard too low)");
    }
}

There are some checks (e.g. verify_adequate_memory_per_shard) that cannot be disabled without using developer_mode. scyllamgr_setup runs Scylla with 1 shard and 500M memory, which is too low for this check to pass.

So the question here is whether it's OK to increase resources used by SM DB created by scyllamgr_setup, or should we ask core team for a way of disabling those checks without developer_mode?

Note that SM does not really relay on its DB performance - it's there mostly for keeping tasks progress and definitions, so it does not need as many resources as regular Scylla cluster. Also note that increasing SM memory consumption might be annoying from cloud/monitoring/operator perspective.

cc: @mykaul @karol-kokoszka @charconstpointer

[Michal-Leszczynski]

But maybe it's OK for scyllamgr_setup to create Scylla in developer_mode (highlight it in the script and docs) as user can always make their own setup and just specify it in scylla_manager.yaml.

[mykaul]

Any checks you need some flag to be skipped - file an issue and I'm sure @syuu1228 will be able to evaluate and assist if it's feasible or not to skip.

[mykaul]

Note that SM does not really relay on its DB performance - it's there mostly for keeping tasks progress and definitions, so it does not need as many resources as regular Scylla cluster. Also note that increasing SM memory consumption might be annoying from cloud/monitoring/operator perspective.

Long term, let's not discuss it here - SM should probably kept its database records in the cluster it was managing...