Open asyslinux opened 2 years ago
you have a few different issues here:
1) i wouldn't run this as root but if it makes things easier for you then for testing that's ok - I would instead ensure your user is in the kvm group that way you don't need to mess w/it - the install script checks for this but if you built from source than it wouldn't have
2) by default 'ops run' will and 'ops pkg load' will build on demand the image and store them in ~/.ops/images/ but I don't think the 'image create' for proxmox will build on demand and instead expects you to already have a local image (in this case /root/.ops/images/test.img ) - so if you try again now that that image is there you should get further along with promox - let me know how that goes - there is more info here on that: https://docs.ops.city/ops/proxmox
3) by default ops will use user-mode networking which nats a single port - that works for most dev use-cases and I wouldn't recommend going down the bridge route until you're comfortable with the other options - more info on setting up a bridge/taps here: https://docs.ops.city/ops/networking#bridged-network
if your goal is to go through proxmox we can use this issue for that - now that you have the test.img you should be able to get further along with that - let me know
Thanks for quick reply.
Yes, I prefer to use Proxmox instead of other variants, but i need in bridged networking or other solution.
P.S. root user i use only for testing.
I want create multiple haproxy unikernel machines + several unikernel machines with custom golang app. This golang app working with remote redis servers. For all of this I need a shared local area network, haproxy used for load balancing requests. How do it without bridged networking, i don`t know. Yes I have multiple physical servers in LAN too.
Ok lets try to work with ProxMox and onprem test.img:
Current config.json:
{
"Dirs": ["static"],
"NameServers": ["172.21.1.50"],
"RunConfig": {
"Imagename": "test",
"InstanceName": "1000",
"Memory": "2G",
"VolumeSizeInGb": 1,
"Ports": ["8288"],
"Background": false,
"Verbose": true,
"ShowErrors": true,
"ShowWarnings": true
}
}
root@my:~/go/src/test#ops image create test -c config.json
100% |████████████████████████████████████████| [1s:0s]
onprem image '/root/.ops/images/test.img' created...
root@my~/go/src/test# ls -la /root/.ops/images/test.img
-rw-r--r-- 1 root root 22177280 Aug 28 12:44 /root/.ops/images/test.img
root@my:~/go/src/test# ops image create test -t proxmox -c config.json
100% |████████████████████████████████████████| [1s:0s]
proxmox image 'test' created...
root@my:~/go/src/test# ops instance create test.img -t proxmox
root@my:~/go/src/test# ops image list -t proxmox
+----+------+------+
| ID | NAME | SIZE |
+----+------+------+
root@my:~/go/src/test# ops instance list -t proxmox
+----+------+--------+--------+---------+
| ID | NAME | MAINIP | STATUS | IMAGEID |
+----+------+--------+--------+---------+
root@my:~/go/src/test# ops instance start 1000 -t proxmox
root@my:~/go/src/test#
All is empty, nothing errors, in ProxMox web interface no new VMs.
ok - so for promox you should not be trying to set up the networking - that's only if you were trying to run your own infrastructure - the way ops works is that it interacts with the various APIs that clouds and hypervisors such as proxmox export - so instead of you setting up the bridge manually proxmox will do that for you (and manage it)
there are 3 env vars that need to be set for proxmox as shown here, https://docs.ops.city/ops/proxmox API_URL, TOKEN_ID, and SECRET - were you able to set those before issuing those commands? also it looks like the image create might've worked - do you see anything new in the storage from the web ui?
ops will integrate directly against proxmox's api - you can verify those env vars should be working by grabbing a version like so:
#!/bin/sh
curl -k \
-H "Authorization: PVEAPIToken=tokenid=secret" \
https://ip:8006/api2/json/pve/version
Yes, 3 env vars is set correctly, if not set, ops says error (i temporary removed env vars):
root@my:~/go/src/test# ops image create test -t proxmox -c config.json
100% |████████████████████████████████████████| [0s:0s]
TOKEN_ID is not set
Its working, but You say use this url for check - https://ip:8006/api2/json/pve/version - this url gives {"data":null}, currently on my proxmox this url is working ok - https://ip:8006/api2/json/version without /pve/ sublocation , examples:
root@my:~/go/src/test# curl -k -H 'Authorization: PVEAPIToken=myuser=myuuid' https://127.0.0.1:8006/api2/json
{"data":[{"subdir":"version"},{"subdir":"cluster"},{"subdir":"nodes"},{"subdir":"storage"},{"subdir":"access"},{"subdir":"pools"}]}
root@my:~/go/src/test# curl -k -H 'Authorization: PVEAPIToken=myuser=myuuid' https://127.0.0.1:8006/api2/json/version
{"data":{"release":"7.2","version":"7.2-4","repoid":"ca9d43cc"}}
root@my:~/go/src/test# curl -k -H 'Authorization: PVEAPIToken=myuser=myuuid' https://127.0.0.1:8006/api2/json/storage
{"data":[{"content":"backup,vztmpl,rootdir,images,snippets,iso","digest":"803f59ff309a8b2f96f1037b8850f7fdc5e8558f","path":"/var/lib/vz","storage":"local","prune-backups":"keep-all=1","type":"dir"},{"storage":"data","path":"/data/vz","digest":"803f59ff309a8b2f96f1037b8850f7fdc5e8558f","content":"backup,rootdir,vztmpl,snippets,iso,images","shared":0,"type":"dir","prune-backups":"keep-all=1"}]}
In proxmox web interface i didn`t find any images, in proxmox folders same /var/lib/vz too.
May be problem in changed urls in late versions of proxmox? Because ops say no errors on every operation and do nothing, like as magic.
so it looks like there might be a hard-coded default
this was tested on proxmox-ve 7.1 - i do see that 7.2 is out
looking at api docs https://pve.proxmox.com/pve-docs/api-viewer/index.html#/nodes/{node}/qemu it appears that it refers to the cluster node name
that's all currently hard-coded so
if you are familiar w/go you could try just substituting or stripping those out and re-building to see if that works for you?
➜ proxmox git:(master) ✗ grep -R pve *
proxmox_image.go: req, err := http.NewRequest("POST", p.apiURL+"/api2/json/nodes/pve/storage/local/upload", &b)
proxmox_image.go: req, err := http.NewRequest("GET", p.apiURL+"/api2/json/nodes/pve/storage/local/content", nil)
proxmox_instance.go: req, err := http.NewRequest("POST", p.apiURL+"/api2/json/nodes/pve/qemu", bytes.NewBufferString(data.Encode()))
proxmox_instance.go: req, err := http.NewRequest("POST", p.apiURL+"/api2/json/nodes/pve/qemu/"+vmid+"/config", bytes.NewBufferString(data.Encode()))
proxmox_instance.go: req, err := http.NewRequest("GET", p.apiURL+"/api2/json/nodes/pve/qemu", nil)
proxmox_instance.go: req, err := http.NewRequest("DELETE", p.apiURL+"/api2/json/nodes/pve/qemu/"+instanceID, nil)
proxmox_instance.go: req, err := http.NewRequest("POST", p.apiURL+"/api2/json/nodes/pve/qemu/"+instanceID+"/status/start", nil)
proxmox_instance.go: req, err := http.NewRequest("POST", p.apiURL+"/api2/json/nodes/pve/qemu/"+instanceID+"/status/stop", nil)
if that does work - then we can modify this to use an env var and use that as the backup default?
Ok, i am familiar with go, write any software, give me pls 1-2 days, i come to work and do it, then reply to You. I think additional customizable env var be good solution for different proxmox setups and changeable urls. Thanks, I should have looked at the ops sources myself earlier, just the lack of errors from ops confused me.
Now I can look images (ops image list -t proxmox):
root@my:~/go/src/test# ops image list -t proxmox
+-----------------------------------------------------+----------------------------------------+--------------+
| ID | NAME | SIZE |
+-----------------------------------------------------+----------------------------------------+--------------+
| local:100/vm-100-disk-0.qcow2 | vm-100-disk-0.qcow2 | 34359738368 |
+-----------------------------------------------------+----------------------------------------+--------------+
| local:200/vm-200-disk-0.qcow2 | vm-200-disk-0.qcow2 | 34359738368 |
+-----------------------------------------------------+----------------------------------------+--------------+
| local:225/vm-225-disk-0.qcow2 | vm-225-disk-0.qcow2 | 34359738368 |
+-----------------------------------------------------+----------------------------------------+--------------+
| local:250/vm-250-disk-0.qcow2 | vm-250-disk-0.qcow2 | 34359738368 |
+-----------------------------------------------------+----------------------------------------+--------------+
| local:500/vm-500-disk-0.qcow2 | vm-500-disk-0.qcow2 | 34359738368 |
+-----------------------------------------------------+----------------------------------------+--------------+
| local:iso/debian-11.2.0-amd64-netinst.iso | debian-11.2.0-amd64-netinst.iso | 396361728 |
+-----------------------------------------------------+----------------------------------------+--------------+
| local:vztmpl/debian-11-standard_11.0-1_amd64.tar.gz | debian-11-standard_11.0-1_amd64.tar.gz | 243431756 |
+-----------------------------------------------------+----------------------------------------+--------------+
root@my:~/go/src/test# ops --show-debug image create test -t proxmox -c config.json
100% |████████████████████████████████████████| [1s:0s]
{"data":null,"errors":{"filename":"wrong file extension"}}
proxmox image 'test' created...
I try change name in config.json:
"RunConfig": {
"ImageName": "test.iso",
But every time, when i start ops image create, then ops built image without extension, and then this image can not be uploaded to proxmox.
Ok, i hardcoded iso extension in func (p *ProxMox) CreateImage(ctx *lepton.Context, imagePath string) error {
fw, err = w.CreateFormFile(fieldName, "test.iso")
And successfully uploaded image.
root@my:~/go/src/test# ops image create test -t proxmox -c config.json
100% |████████████████████████████████████████| [0s:0s]
{"data":"UPID:my:002727BC:2DE1048C:630C9D7F:imgcopy::root@pam!id:"}
proxmox image 'mytest' created...
Create instance
root@my:~/go/src/test# ops instance create mytest -t proxmox -c config.json
root@my:~/go/src/test#
Instance is created, but iso image do not connected automatically. Ok I add CD/DVD drive manually in proxmox and connect test.iso image.
Start machine, and have an error: Could not read from CDROM (code 0004) This onprem image is not bootable as iso image :(
I can take it and improve the сode for work with proxmox, fix a bug with the extension, add custom storage selection CloudConfig option, automatically mount the right image when creating an instance, and I can also make other improvements and features, then I will send a pull request.
But what to do with the non-bootable image? I think this is strange - use and upload onprem image as iso image to proxmox. Also i try use onprem image for booting as raw harddisk, its not worked too.
I figured it out, I was able to boot the image as an attached hard drive based on the virtio driver. On a virtual machine with q35 type. And bridged networking also works with virtio network driver, bridge networking is built by proxmox.
root@my:/var/lib/vz/images/103# ping 172.21.1.75 PING 172.21.1.75 (172.21.1.75) 56(84) bytes of data. 64 bytes from 172.21.1.75: icmp_seq=1 ttl=255 time=0.261 ms 64 bytes from 172.21.1.75: icmp_seq=2 ttl=255 time=0.197 ms 64 bytes from 172.21.1.75: icmp_seq=3 ttl=255 time=0.180 ms
Ok, can i start development improvements(including add more features) to proxmox part code of ops? Then i`ll be send pull requests. I want to write full support of proxmox for ops.
you might be hitting a different issue with the .iso suffix as that looks like it should be getting set here:
https://github.com/nanovms/ops/blob/master/proxmox/proxmox_image.go#L67
the image can't be ran as a cdrom - when you create the instance with ops it attaches as a virtio disk https://github.com/nanovms/ops/blob/master/proxmox/proxmox_instance.go#L94 as you found out - ops will set up all of this for you - the intention is that the end-user shouldn't be doing anything manually at all
yes if you want to work on this just drop each feature/bug into a sep. pr please - for instance the default node name set by env var could be one
i fixed the .iso extension issue - at some point in the past half year or so we had removed all extensions which broke that but proxmox requires it; also for whatever reason we never put in code to mv the 'iso' to disk so put that in as well - https://github.com/nanovms/ops/pull/1362
Thanks, later i look into api of proxmox and test all of possibilities with images (including upload, replace, attach, etc), because in proxmox normally need to create .raw images, not upload .iso . But let's see later what options are possible.
this https://github.com/nanovms/ops/pull/1363 fixes the case where an image isn't built locally yet before trying to upload to proxmox - it ensures the image is built first which is the default behavior for all other targets
I created new pull request with additional checks of returned empty data from proxmox api: #1364 Now is only checked empty data, support of extendeded errors from proxmox api will be coded in next steps.
Later, when i will add configurable storage names, then I will add check for errors for this two calls functions: https://github.com/nanovms/ops/blob/master/proxmox/proxmox_instance.go#L89 p.addVirtioDisk(ctx, nextid, imageName) p.movDisk(ctx, nextid, imageName)
I made configurable storages, bridge interface(with check of configured bridge in proxmox) via CloudConfig and config.json. I haven't created a pull request yet, because I would like to know if such variable names in the CloudConfig structure and their location are suitable for a configuration of instances in ProxMox via config.json . I also made support for reverting to local/local-lvm,vmbr0 for backwards compatibility.
I created a config.CloudConfig.BridgeName0 (Not BridgeName), because later I think we need to have support of minimum two interfaces (if nanos and ops have planned support of more than one connected network interface). For example in my case I need in the future have ability to use two bridge interfaces from ProxMox in case of using haproxy in nanovms (first for internet, second for local area network and load balancing/high availability to backend apps in local area network).
Also I added instanceName from config.RunConfig.InstanceName (You made this in RunConfig part of config.json) and now can be created instances with timestamp support from cmd/cmd_instance.go. I do not want broke any backward-compatibility, and uses config.RunConfig.InstanceName as is. Next I planned to add support for configurable Machine Type(i440fx/q35)/CPU/Cores/Memory/Onboot in ProxMox from config.json (But where? In RunConfig or CloudConfig).
In general, I need to know where to place which options(in RunConfig/CloudConfig) for ProxMox (StorageName, IsoStorageName, BridgeName0, CPUs, Cores, Memory, Onboot, Machine Type, etc)
Can I create a pull request? Thanks. In which case, I can always rewrite code.
Example of code:
instanceName := config.RunConfig.InstanceName
imageName := config.CloudConfig.ImageName
storageName := config.CloudConfig.StorageName
isoStorageName := config.CloudConfig.IsoStorageName
bridgeName0 := config.CloudConfig.BridgeName0
if bridgeName0 == "" {
bridgeName0 = "vmbr0"
}
err = p.CheckBridge(bridgeName0)
if err != nil {
return err
}
data := url.Values{}
data.Set("vmid", nextid)
data.Set("name", instanceName)
data.Set("net0", "model=virtio,bridge="+bridgeName0)
Example of config.json
{
"Dirs": ["static"],
"NameServers": ["172.21.1.1"],
"CloudConfig": {
"BridgeName0": "vmbr1",
"ImageName": "mytest",
"IsoStorageName": "local",
"StorageName": "local-lvm"
},
"RunConfig": {
"Accel": true,
"InstanceName": "mytest",
"ImageName": "mytest",
"Memory": "2G",
"IPAddress": "172.21.1.75",
"NetMask": "255.255.255.0",
"Gateway": "172.21.1.1",
"Verbose": true,
"ShowErrors": true,
"ShowWarnings": true
}
}
I think, config for real production in case of ProxMox is will be like this (minimum): What do you think for the future, would such a config suit us?
{
"Dirs": ["static"],
"NameServers": ["172.21.1.1"],
"CloudConfig": {
"Arch": "x86_64", // or "Arch": "aarch64" or automatic detected by proxmox.
"Machine": "q35", // or "Machine": "pc" by default,
"CPUs": 2, // or "Sockets": 2, or "CPUs": 1 or "Sockets": 1 by default
"Cores": 8, // or "Cores": 1 by default
"Numa": true, // or "Numa": false by default
"Memory": "512M", // or "Memory": "2G" (convertible later to integer in megabytes) or 512M by default,
"BridgeName0": "vmbr0", // or "BridgeName": "vmbr0" main interface (linked with net0 in ProxMox for instance) or "vmbr0" by default
"BridgeName1": "vmbr1", // (if will have support for secondary network interface in nanos/ops and RunConfig)
"ImageName": "mytest", // or get from cmd
"IsoStorageName": "local", // or "local" by default
"StorageName": "local-lvm", // or "local-lvm" by default
"Protection": false, // protection of disk images, if true, prevent delete or replace image, false by default
"Onboot": true // and true by default
},
"RunConfig": {
"Accel": true,
"InstanceName": "mytest", // (for backward compatibility, instance name in Proxmox be leaved here), // or get from cmd
"ImageName": "mytest", // or get from cmd
"IPAddress0": "1.2.3.4", // or "IPAddress": "1.2.3.4", (for backward compatibility)
"NetMask0": "255.255.255.0", // or "NetMask": "255.255.255.0", (for backward compatibility)
"Gateway0": "1.2.3.1", // or "Gateway": "1.2.3.1", (for backward compatibility)
"IPAddress1": "172.21.1.75", // (if will have support for secondary network interface in nanos/ops and RunConfig)
"NetMask1": "255.255.255.0", // (if will have support for secondary network interface in nanos/ops and RunConfig)
"Verbose": true,
"ShowErrors": true,
"ShowWarnings": true
}
}
so some thoughts (as someone who doesn't work day to day with proxmox):
1) most of our targets (like gcp or aws) don't set bridge name - instead they set a vpc which typically but not always means "same routable network" so I suppose I'm ok with setting that in lieu of this
2) the static ip addressing and netmask and gateway should not be set by default - instead dhcp is used - if an ip address is set that implies that one wants it static
4) in general all of machine/core count/etc. stuff I'd throw into cloudconfig - some of this exists already for other target platforms (and it's something that will probably continue to be revised in the future
Hello, I try to use ops with local ProxMox 7.2 and Ops 0.1.32, and it isn`t working:
System information:
Access rigths is full on token in ProxMox. Image can not be build, nothing any errors from ops.
In onprem mode is all starting ok, excluding bridge - is non-functional:
Machine is not available through host bridge.
I try to add "TapName": "tap0" or "TapName": "tap75i0" or any other name, but in this case of use onprem I have additional trouble (Not able to create tap, but modprobe tap is ok, and on this server I have so much other tap* interfaces):
Only all working through exposed port. Without TapName,Bridged,BridgeName,IPAddress,Gateway,NetMask options:
But this is can not be used for production. Please, can anyone help? May be i missing something or do something wrong? Thanks.