canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.38k stars 931 forks source link

lxc init command hangs when executed as process from some language #6856

Closed LukasHanusek closed 4 years ago

LukasHanusek commented 4 years ago

lxc info:


config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIICDDCCAZOgAwIBAgIRAL85cLrgveNtvhxlSfwu+LcwCgYIKoZIzj0EAwMwNzEc
    MBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEXMBUGA1UEAwwOcm9vdEBOZW9u
    Q2xvdWQwHhcNMjAwMTMxMTc1NjEzWhcNMzAwMTI4MTc1NjEzWjA3MRwwGgYDVQQK
    ExNsaW51eGNvbnRhaW5lcnMub3JnMRcwFQYDVQQDDA5yb290QE5lb25DbG91ZDB2
    MBAGByqGSM49AgEGBSuBBAAiA2IABDdNXEOXL3IUzf8g2jSocJB6C5/3LOximl1L
    Id0VBvmkTzGXbcS/2ufvFD752igIXdooKffT23YD5wVAMzyLl8D8KEVM6nNIqnnQ
    vSAxp7ylDDnSMKppzJo8IYu0jkmR5KNjMGEwDgYDVR0PAQH/BAQDAgWgMBMGA1Ud
    JQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwLAYDVR0RBCUwI4IJTmVvbkNs
    b3VkhwSQTHhGhxAqAQT4AZKARwAAAAAAAAACMAoGCCqGSM49BAMDA2cAMGQCMCyw
    vej6PpPVGbUcTDoG8pwAdinAEsSBUz+xUJYcFV34h1rdnT8FStMFy6sQU6W2eQIw
    EU7SzjhKvHCdmBVazA6qmqvwvJdZRw6uYnAftGMVv4jtKhH8bv3iG2USR6oI9a0U
    -----END CERTIFICATE-----
  certificate_fingerprint: 1a5362b08107caa038b86827b28d088d12d97a77b98aeb509aca5ca93ddac49e
  driver: lxc
  driver_version: 3.2.1
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    netnsid_getifaddrs: "false"
    seccomp_listener: "false"
    seccomp_listener_continue: "false"
    shiftfs: "false"
    uevent_injection: "true"
    unpriv_fscaps: "true"
  kernel_version: 4.19.0-6-amd64
  lxc_features:
    cgroup2: "false"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    seccomp_notify: "true"
  project: default
  server: lxd
  server_clustered: false
  server_name: NeonCloud
  server_pid: 8029
  server_version: "3.20"
  storage: btrfs
  storage_version: "4.4"

Issue description

I am trying to automate LXC container creation process. So I generated LXC .sh script to create a new LXC container. It works when I execute it manually from ssh. However when I try to run this script from Java programming language, it hangs on "lxc init" for some reason and I cannot figure out why.

The script:

#!/bin/sh
echo 1
. /etc/profile.d/apps-bin-path.sh
echo 2
lxc init 'images:debian/stretch' LXC8
echo 3
lxc network attach lxdbr0 LXC8 eth0
lxc config device set LXC8 eth0 ipv4.address 10.10.2.15
lxc config set LXC8 limits.cpu 1
lxc config set LXC8 limits.cpu.allowance 12%
lxc config set LXC8 limits.memory 1024MB

How I tried to execute it (from Java language):

ProcessBuilder pb = new ProcessBuilder("/home/nc/tmp/myscript.sh");
Process p = pb.start();
p.waitFor();

And with debug (BufferedReader.readLine()), the last thing I see is "2" which is the "echo 2" right before lxc init, then it hang indefinitely.

Is there anything that might be blocking the execution of lxc init from Java ? It is very weird because the Java is only triggering the script.sh and the commands are not directly executed via java process builder.

All other LXC commands (when executed from java) are working perfectly, just this one is not.

LukasHanusek commented 4 years ago

Tried to execute the same script via PHP so to confirm that it is not just Java's problem, I get exactly the same behavior, it hang indefinitely on lxc init with PHP as well:

<?php
$descriptor = [
    0 => ["pipe", "r"],
    1 => ["pipe", "w"]
];
$proc = proc_open("/home/nc/tmp/myscript.sh", $descriptor, $pipes);
echo stream_get_contents($pipes[1]);
fclose($pipes[1]);
?>
LukasHanusek commented 4 years ago

Fixed by immediately closing outputstream after process start:

        Process p = pb.start();
        p.getOutputStream().close();
ibrahimmenem commented 4 years ago

If using python with paramiko setting get_pty=True will fix it http://docs.paramiko.org/en/stable/api/client.html#paramiko.client.SSHClient.exec_command

aartiPl commented 1 year ago

Hi! I have the same problem on Ubuntu 20.04 (in Vagrant). The lxc process hangs forever when started from ProcessBuilder. In my case, it hangs for "lxc launch". The solution from @LukasHanusek works, but IMHO, the problem still needs to be solved. I am creating a generic tool for executing different commands, and I really don't understand why I should close the output stream, just after starting ProcessBuilder. It doesn't sound as the correct solution, but rather just a workaround.

Is it possible to reopen this ticket, or should I create a new one?

tomponline commented 1 year ago

lxc init reads config from stdin, so if you are not passing config in you need to ensure stdin is closed.

aartiPl commented 1 year ago

I have just updated my comment: in my case, it happens also during "lxc launch". So AFAIK it doesn't read input, am I wrong?

aartiPl commented 1 year ago

And in fact also "lxc init" can be correctly executed without providing stdin. I think that in such a case it should not block on reading stdin anyway, and just check if the input is provided. In my case "lxc init" is not a problem - I am feeding "lxc init" with configuration. But "lxc launch" also blocks for following arguments:

lxc launch -p development ubuntu:20.04 my-server

tomponline commented 1 year ago

lxc launch also reads from stdin as it is a convenience combination of lxc init and lxc start.

Closing or not providing stdin will prevent these commands from blocking.

aartiPl commented 1 year ago

Ok, just for the record. Here is a minimal test case:

import java.io.IOException;
import java.util.concurrent.TimeUnit;

public final class Main {
    public static void main(String[] args) throws IOException, InterruptedException {
        System.out.println("Program started");

        Process process = new ProcessBuilder("bash", "-c", "lxc launch -p development ubuntu:20.04 my-server")
                .redirectInput(ProcessBuilder.Redirect.PIPE)
                .redirectOutput(ProcessBuilder.Redirect.PIPE)
                .start();

        //process.getOutputStream().close();
        process.waitFor(10, TimeUnit.MINUTES);

        System.out.println("Program finished");
    }
}

You can start it like that:

javac Main.java && java Main

So if the intention is to write to lxc stdin from a Java application, then there should be used following code to do it:

OutputStreamWriter writer = new OutputStreamWriter(process.getOutputStream());

and the required data should be written using writer, or if nothing should be written, we can just close the stream.

If the intention is to provide data from Java process stdin, then the following form should be used in ProcessBuilder:

.redirectInput(ProcessBuilder.Redirect.INHERIT)

It's indeed not a problem of LXC. The exact result you will get if the bash command is read NAME. Maybe one thing in LXC is slightly misleading: when you start lxc launch from the command line, it never hangs, waiting for user input. But when it is started from ProcessBuilder it hangs forever.