PVE使用

[cndaqiang]

没事别更新系统

空间被占用完了 重启后不能开机了

[cndaqiang]

下载

https://www.proxmox.com/en/downloads 使用Torrent下载速度快 也可以从清华下https://mirrors.tuna.tsinghua.edu.cn/proxmox/iso/

刻录U盘

开机启动

需要鼠标操作，只用键盘不可以

安装盘要大

3G报错太小 USB插的U盘，要提前格式化成一块ext分区才可以使用 安装到32G的USB2.0U盘中, 前50%很快, 23min后,安装进度61%,终止安装,累了 安装到16G的USB3.0U盘,3.5min就安装到63%,还是要用USB3.0安装

最后的硬盘占用情况

安装配置ip

pve会自动检测到连接路由器的网卡,设置好ip. 我这里不想用路由器, 把笔记本网络用网线共享给PVE主机 笔记本端网络配置，共享网络 PVE也可以自动检测到共享的网络，获取一个内网ip 以后配置其他机器也可以这样共享

debian安装法

先安装debian 移除/etc/hosts中的127.0.1.1选项，添加自己的ip和主机名 tuna

#
echo "deb https://mirrors.tuna.tsinghua.edu.cn/proxmox/debian buster pve-no-subscriptio" > /etc/apt/sources.list.d/pve-no-subscription.list

or 官方

#此方式没法更新,难道是https的原因？
echo "deb http://download.proxmox.com/debian/pve buster pve-no-subscription" > /etc/apt/sources.list.d/pve-install-repo.list

gpg

wget http://download.proxmox.com/debian/proxmox-ve-release-6.x.gpg -O /etc/apt/trusted.gpg.d/proxmox-ve-release-6.x.gpg
chmod +r /etc/apt/trusted.gpg.d/proxmox-ve-release-6.x.gpg  # optional, if you have a non-default umask

更新

apt update && apt full-upgrade

安装

apt install proxmox-ve postfix open-iscsi

移除

apt remove os-prober

网页访问

[cndaqiang]

开机

一定要使用https打开,http打不开网页

https://10.42.0.85:8006/

登录

网页登录shell,或者平面登录shell 用户名root,密码开机时设置

PVE主机设置dns

root@pve:~# nano  /etc/resolv.conf 

设置为114.114.114.114

nameserver 114.114.114.114

改源

https://mirrors.tuna.tsinghua.edu.cn/help/proxmox/

cd /etc/apt/sources.list.d/
echo "deb https://mirrors.tuna.tsinghua.edu.cn/proxmox/debian buster pve-no-subscription" > /etc/apt/sources.list.d/pve-no-subscription.list
mv pve-enterprise.list pve-enterprise.list.bak

debian

mv /etc/apt/sources.list /etc/apt/sources.list.bak
 vi /etc/apt/sources.list

修改为

# 默认注释了源码镜像以提高 apt update 速度，如有需要可自行取消注释
deb https://mirrors.tuna.tsinghua.edu.cn/debian/ buster main contrib non-free
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ buster main contrib non-free
deb https://mirrors.tuna.tsinghua.edu.cn/debian/ buster-updates main contrib non-free
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ buster-updates main contrib non-free
deb https://mirrors.tuna.tsinghua.edu.cn/debian/ buster-backports main contrib non-free
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ buster-backports main contrib non-free
deb https://mirrors.tuna.tsinghua.edu.cn/debian-security buster/updates main contrib non-free
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian-security buster/updates main contrib non-free

更新

apt update

卸载vim

apt remove vim*
apt install vim

其他

安装这个

apt install ifupdown2

为每个网卡创建一个桥,不知道为什么，好像虚拟机只能使用网桥??

[cndaqiang]

无线网络

apt  install wireless-tools
apt install wpasupplicant

使用ip addr查看网卡名称 启动无线网卡，进行扫描

ifup wlp2s0 
iwlist scan

扫描

root@pve:/etc/network# iwlist scanning | grep ESSID
lo        Interface doesn't support scanning.

vmbr0     Interface doesn't support scanning.

enp3s0    Interface doesn't support scanning.

                    ESSID:"CMCC-34pQ"
                    ESSID:"MI 9"
                    ESSID:"HUAWEI-CPLK3B_HiLink"
                    ESSID:"ChinaNet-tM5S"

连接无密码wifi

 iwconfig wlp2s0 essid "MI 9"

连接加密wifi

cp /etc/network/interfaces /etc/network/interfaces.bak
vi /etc/network/interfaces

添加

 iface wlp3s0 inet dhcp
           wpa-ssid $NETWORK_name
           wpa-psk $HASHED_password

使用wpa_passphrase进行计算得到wifi密码hash形式

(python37) cndaqiang@girl:~$ wpa_passphrase NETWORK_name password
network={
    ssid="NETWORK_name"
    #psk="password"
    psk=30ce1f56b3bdf3e5f61faab80a235677ab1c405fd312236d803d61170f70ae3b
}

使用明文也可以

启动连接

root@pve:/etc/apt/sources.list.d# ifup wlp2s0
root@pve:/etc/apt/sources.list.d# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
    link/ether b8:97:5a:a0:e3:c4 brd ff:ff:ff:ff:ff:ff
3: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0d:f0:c4:ae:41 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.45/24 brd 192.168.1.255 scope global dynamic wlp2s0
       valid_lft 259198sec preferred_lft 259198sec
    inet6 2409:8a3c:5f33:2890:20d:f0ff:fec4:ae41/64 scope global dynamic mngtmpaddr 
       valid_lft 259190sec preferred_lft 172790sec
    inet6 fe80::20d:f0ff:fec4:ae41/64 scope link 
       valid_lft forever preferred_lft forever
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b8:97:5a:a0:e3:c4 brd ff:ff:ff:ff:ff:ff
    inet 10.42.0.85/24 brd 10.42.0.255 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::ba97:5aff:fea0:e3c4/64 scope link 
       valid_lft forever preferred_lft forever

重启服务

/etc/init.d/networking restart

[cndaqiang]

挂载硬盘

好像不能网页端挂载修改, 和linux一样,要使用命令挂载 挂载方法参考https://cndaqiang.github.io//2017/10/11/ubuntu-disk/ 为了利用空间,我在终端把Mint的系统硬盘挂载了，并在内部新建了一个comm目录用挂载(这个目录也可以是任意的目录, 以及ntfs分区) 挂载后查看

挂载另一外U盘

root@pve:~# blkid
/dev/sdc1: LABEL="Lenovo" UUID="e2a7d75d-bda6-4ce4-80a0-c4dbcccc7192" TYPE="ext4" PARTUUID="79960bc9-01"

echo "UUID=e2a7d75d-bda6-4ce4-80a0-c4dbcccc7192 /data/Lenovo ext4 defaults,errors=remount-ro 0 1" >> /etc/fstab 
mkdir -p /data/Lenovo

[cndaqiang]

安装示例:爱快

上传安装镜像ISO

进入一个数据中心，上传ikuai 也可以直接复制到template/iso/

直接创建即可,可以选择存储的硬盘

32位爱快开机花屏，显卡选择vm模式 参考https://bbs.ikuai8.com/thread-98366-1-1.html 网卡选择能连通笔记本的网卡所在的网桥 启动后从控制台

设置lan口地址为10.42.0.86 通过网页访问 http://10.42.0.86/ 账户密码admin,admin

[cndaqiang]

开机启动脚本

使用systemctl设置开机启动 https://www.yigmx.com/624.html

cat > /etc/systemd/system/rc-local.service <<EOF
[Unit]
Description=/etc/rc.local
ConditionPathExists=/etc/rc.local
[Service]
Type=forking
ExecStart=/etc/rc.local start
TimeoutSec=0
StandardOutput=tty
RemainAfterExit=yes
SysVStartPriority=99
[Install]
WantedBy=multi-user.target
EOF

cat > /etc/rc.local <<EOF
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
# bash /root/bindip.sh
exit 0
EOF

chmod +x /etc/rc.local
systemctl enable rc-local
systemctl start rc-local.service

修改/etc/rc.d添加启动命令,如自动连接无线网卡

ifup wlp2s0

[cndaqiang]

目前没有办法让有线网络连上路由器,也不能把无线网络共享给虚拟机

暂时采用ssr方式连通无线有线网 笔记本 ->无线-> ssr->PVE->有线网卡->虚拟机 就可以访问了

https://10.42.0.85:8006

服务器添加到/etc/rc.local

/root/shadowsocksr-manyuser/shadowsocks/server.py -c /root/shadowsocksr-manyuser/config.json -d start

设置路由链路

apt install net-tools
#默认网关无线网络的网关
route add default gw 192.168.1.1
#内网网关有线网桥的网关
route add -net 10.42.0.0 netmask 255.255.255.0 dev vmbr0

[cndaqiang]

报错

上传iso报错错误 501: upload failed 因为不是ISO文件的原因

[cndaqiang]

PVE建立NAT网桥

参考Proxmox VE 踩坑记录

PVE建立网桥

vi /etc/network/interfaces 添加

auto vmbr2
iface vmbr2 inet static
    address 10.0.0.254
    netmask 255.255.255.0
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    post-up echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j MASQUERADE

重启网络

/etc/init.d/networking restart

NAT固定相应ip

网关设置为PVE的网桥ip

例如想将宿主机vmbr0的80端口的tcp连接转发到10.0.0.102的80端口上：

iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 80 -j DNAT --to 10.0.0.102:80

如果想保存转发规则，使之重启后依然有效，则需要在/etc/network/interfaces相应位置加入

post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 80 -j DNAT --to 10.0.0.102:80
post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 80 -j DNAT --to 10.0.0.102:80

[可选]创建一个ikuai的虚拟机用于dhcp

设置ikuai的lan口为10.0.0.100 使用任一虚拟机连山ikuai配置 但是ikuai本身不能设置网关，也就不能上网

[可选]LEDE进行dhcp

vi /etc/config/network

/etc/network/restart

注kooshare的vmdk格式的LEDE,重置一下再使用,不然各种问题

[cndaqiang]

虚拟机设备操作

删除硬盘

1.分离 2.删除 3.把分离的磁盘加回来 编辑添加

添加已有虚拟磁盘

vi /etc/pve/qemu-server/<ID>.conf

依此添加

unused0: D3PVE:100/vm-100-disk-0.qcow2
unused1: D3PVE:100/vm-100-disk-1.qcow2
...

不同格式的硬盘:Raw vs qcow2 vs vmdk

空白情况下空间占用(总容量32G)

-rw-r----- 1 root root 34365243392 Aug 30 18:45 vm-100-disk-0.qcow2
-rw-r----- 1 root root   144834560 Aug 30 18:50 vm-100-disk-1.vmdk
-rw-r----- 1 root root 34359738368 Aug 30 18:46 vm-100-disk-2.raw

虚拟机内硬盘第一次初始化格式化时, vmdk的速度非常慢，qcow2和raw都是秒格式化

直通U盘

使用U盘安装win10

Linux下的多块光驱

使用srN挂载即可

[root@localhost cndaqiang]# ll /dev/cdrom 
lrwxrwxrwx. 1 root root 3 9月   1 09:09 /dev/cdrom -> sr0
[root@localhost cndaqiang]# ls /dev/sr*
/dev/sr0  /dev/sr1

[cndaqiang]

NFS共享

参考从0搭建Centos7 计算集群 Debian Linux安装NFS 在PVE主机开启NFS共享 各虚拟机挂载该NFS硬盘，实现数据互通

虽然能实现数据互通，但是访问速度实在是太慢了，解压个tar包都要半天，更别提安装程序了，还是以分享数据为主，毕竟不是IB网络

apt install nfs-common nfs-kernel-server

共享目录

mkdir /home/D3/NFS
chmod a+w /home/D3/NFS

添加配置文件vi /etc/exports

/home/D3/NFS 10.0.0.0/24(rw,insecure,sync) 10.127.6.47(rw,insecure,sync) 10.60.7.51(rw,insecure,sync) 10.0.100.0/24(rw,insecure,sync) 10.127.6.39(rw,insecure,sync) 192.168.1.0/24(rw,insecure,sync)

其中10.127.6.47是我另一台路由器的地址，路由器下面的主机也可以挂载到这个NFS insecure参数防止mac突然断掉nfs连不上，和报错Operation not permitted 20230908:我的最新配置192.168.192.0/24(rw,no_subtree_check,all_squash,anonuid=1000,anongid=1000) 重启服务，检测

root@pve:/home/D3/PVE/images/100# systemctl restart nfs-server.service 
root@pve:/home/D3/PVE/images/100# showmount -e localhost
Export list for localhost:
/home/D3/NFS 10.0.0.0/24,10.127.6.47

不要用PVE的root创建文件夹,客户端连上后可能因文件夹没有权限不可读写

Mint客户端

apt install nfs-common

开机自动挂载命令见 从0搭建Centos7 计算集群

mac也可以挂载

需要sudo权限

cndaqiang@mac tmp$ mount -o resvport mom:/home/D3/NFS ./nfs
mount_nfs: can't mount /home/D3/NFS from mom onto /Users/cndaqiang/tmp/nfs: Operation not permitted
mount: /Users/cndaqiang/tmp/nfs failed with 1
cndaqiang@mac tmp$ sudo mount -o resvport mom:/home/D3/NFS  /Users/data

Mac挂载建议使用下面的命令,不然会频繁掉线，无法打开nfs中的安装程序，卡死等nfs server mom:/home/D3/NFS: lockd not responding http://www.mauserrifle.nl/linux/osx-and-linux-nfs-shares/

sudo mount_nfs  -P -o nolocks,nosuid mom:/home/D3/NFS /Users/data

mac wifi变ip时可能会出现

mount_nfs: can't mount /home/D3/NFS from mom onto /Users/data: Permission denied

设置固定ip的 避免挂载失败，mac失效,备份脚本在git目录

ln -s /Users/cndaqiang/git /Users/data/code

Windows客户端

控制面板>程序>开启NFS

[cndaqiang]

PVE安装WIN10需要ISCI驱动才能识别硬盘

https://wangye.org/blog/archives/1216/

[cndaqiang]

界面

双击这里的主机名可以打开全屏的noVNC

[cndaqiang]

统一ssh

rm ssh work package code

#ln -s $HOME/../data/ssh ~/ssh
if [ -d ~/ssh ]
then
   rm -rf ~/.ssh
  mkdir ~/.ssh
  cp -r ~/ssh/*  ~/.ssh/
  chmod 600 ~/.ssh/authorized_keys
  chmod 600 ~/.ssh/id_rsa
  chmod 600 ~/.ssh/id_rsa.pub
  chmod 700 ~/.ssh
fi

统一代码工作目录

ln -s $HOME/../data/ssh .
ln -s $HOME/../data/work/ .
ln -s $HOME/../data/package/ .
ln -s $HOME/../data/code/ .

mint系统统一程序

#mint 
ln -s code/mint19/anaconda3 .
#centos 
ln -s code/centos7/anaconda3 .

# added by Anaconda3 5.3.1 installer
# >>> conda init >>>
# !! Contents within this block are managed by 'conda init' !!
if [ -d ~/anaconda3 ]
then
__conda_setup="$(CONDA_REPORT_ERRORS=false '/home/cndaqiang/anaconda3/bin/conda' shell.bash hook 2> /dev/null)"
if [ $? -eq 0 ]; then
    \eval "$__conda_setup"
else
    if [ -f "/home/cndaqiang/anaconda3/etc/profile.d/conda.sh" ]; then
        . "/home/cndaqiang/anaconda3/etc/profile.d/conda.sh"
        CONDA_CHANGEPS1=false conda activate base
    else
        \export PATH="/home/cndaqiang/anaconda3/bin:$PATH"
    fi
fi
unset __conda_setup
fi
# <<< conda init <<<

太卡顿了

[cndaqiang]

添加nfs存储

create storage failed: mkdir /mnt/pve/mom/images: Permission denied at /usr/share/perl5/PVE/Storage/Plugin.pm line 1175. (500)

改服务端配置

192.168.192.0/24(rw,insecure,sync,no_root_squash)

[cndaqiang]

去除订阅信息

sed -i "s/data.status !== 'Active'/false/g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js

[cndaqiang]

图形化界面

#删除原来的企业订阅源
rm /etc/apt/sources.list.d/pve-enterprise.list
#添加官方非订阅免费源
echo 'deb http://download.proxmox.com/debian/pve buster pve-no-subscription' >> /etc/apt/sources.list.d/pve-no-subscription.list
#更新一下源
apt update

选择一个要安装的图形界面

tasksel

安装后重启即可(Xfce是真的朴素....)

切换Xfce为Gnome

update-alternatives --config x-session-manager

[cndaqiang]

远程图形化界面

    # Install new packages
    sudo apt-get install xrdp xorg

    # Add xrdp user to ssl-cert group and reboot
    sudo adduser xrdp ssl-cert
    sudo reboot

[cndaqiang]

直通pci

提示IOMMU not present

vi /etc/default/grub
#modified
#GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream"
vi  /etc/modules
#add 
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

更新

update-initramfs -u -k all
update-grub
reboot

[cndaqiang]

直通硬盘

指定硬盘方式

root@girl:~# ls /dev/disk/by-id
ata-Acer_GT500A_512G_ASG50512A8CLC92002             ata-SanDisk_X400_M.2_2280_128GB_171367422642-part5
ata-Acer_GT500A_512G_ASG50512A8CLC92002-part1       mmc-ED2S5_0x78565563
ata-Acer_GT500A_512G_ASG50512A8CLC92002-part2       mmc-ED2S5_0x78565563-part1
ata-Acer_GT500A_512G_ASG50512A8CLC92002-part3       wwn-0x5001b444a62e2af8
ata-Acer_GT500A_512G_ASG50512A8CLC92002-part4       wwn-0x5001b444a62e2af8-part1
ata-SanDisk_X400_M.2_2280_128GB_171367422642        wwn-0x5001b444a62e2af8-part2
ata-SanDisk_X400_M.2_2280_128GB_171367422642-part1  wwn-0x5001b444a62e2af8-part3
ata-SanDisk_X400_M.2_2280_128GB_171367422642-part2  wwn-0x5001b444a62e2af8-part4
ata-SanDisk_X400_M.2_2280_128GB_171367422642-part3  wwn-0x5001b444a62e2af8-part5
ata-SanDisk_X400_M.2_2280_128GB_171367422642-part4
root@girl:~# qm set 101(虚拟机的ID) -sata1 /dev/disk/by-id/ata-SanDisk_X400_M.2_2280_128GB_171367422642

如果是UEFI的引导，要改bios为OVMF (UEFI), 还要设置启动项为sata(选项>引导顺序)

[cndaqiang]

控制台分辨率

需要安装驱动 https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio 然后bios设置即可

大分辨率后卡了。。。