myml / myml.github.io

myml的个人网站
https://myml.dev
3 stars 0 forks source link

open build service 使用笔记 #36

Open myml opened 1 year ago

myml commented 1 year ago

ssh默认用户名密码 root:opensuse web默认用户名密码 Admin:opensuse

服务源代码目录:/usr/lib/obs 构建脚本目录:/usr/lib/obs/server/build => /usr/lib/build 构建缓存目录:/var/cache/obs/worker 调度日志目录:/srv/obs/log

初始化系统

启动时间过长

官网 oem 镜像在开机时会循环10分钟检查域名,没有域名可以修改 /usr/lib/obs/server/setup-appliance.sh 脚本 get_hostname 函数中的 TIMEOUT 变量,改成 TIMEOUT=3

第二种解决方法是 更改hostname为 obs,然后在/etc/hosts添加

127.0.0.1 obs
127.0.0.1 obs.lan

配置静态IP (DHCP环境忽略此步骤)

网卡配置 /etc/sysconfig/network/ifcfg-eth0

BOOTPROTO='static'
BROADCAST=''
ETHTOOL_OPTIONS=''
IPADDR='192.168.1.3'
MTU='1400'
NAME=''
NETMASK='255.255.255.0'
NETWORK=''
REMOTE_IPADDR=''
STARTMODE='auto'
DHCLIENT_SET_DEFAULT_ROUTE='no'

DNS配置 /etc/sysconfig/network/config

...
NETCONFIG_DNS_STATIC_SERVERS="x.x.x.x"
...

路由配置 /etc/sysconfig/network/routes

default 192.168.1.1 --

启动ssh服务

systemctl enable sshd --now

更新系统

zypper up
myml commented 1 year ago

添加 Deepin 仓库

仓库源项目

项目 Meta

<project name="Deepin:23">
  <title/>
  <description/>
  <person userid="Admin" role="maintainer"/>
  <build>
    <disable/>
  </build>
  <publish>
    <disable/>
  </publish>
  <repository name="codename_main">
    <download arch="x86_64" url="https://pools.uniontech.com/deepin-beige/beige/main" repotype="deb"/>
    <arch>x86_64</arch>
  </repository>
  <repository name="codename_community">
    <download arch="x86_64" url="https://pools.uniontech.com/deepin-beige/beige/community" repotype="deb"/>
    <arch>x86_64</arch>
  </repository>
  <repository name="codename_commercial">
    <download arch="x86_64" url="https://pools.uniontech.com/deepin-beige/beige/commercial" repotype="deb"/>
    <arch>x86_64</arch>
  </repository>
  <repository name="all">
    <path project="Deepin:23" repository="codename_main"/>
    <path project="Deepin:23" repository="codename_community"/>
    <path project="Deepin:23" repository="codename_commercial"/>
    <arch>x86_64</arch>
  </repository>
</project>

项目 Config

Type: dsc
Repotype: debian

构建应用项目

项目 Meta

<project name="linglong-repo">
  <title/>
  <description/>
  <person userid="Admin" role="maintainer"/>
  <repository name="Deepin_23_all">
    <path project="Deepin:23" repository="all"/>
    <arch>x86_64</arch>
  </repository>
</project>

项目 Config


# create initial user
Preinstall: base-passwd
Preinstall: user-setup

# required for preinstall images
Preinstall: perl

ExpandFlags: preinstallexpand
Preinstall: init-system-helpers debianutils dpkg libc-bin dash coreutils diffutils
Preinstall: sed gawk grep gzip debconf bash base-files base-passwd libsystemd0 xz-utils
Preinstall: findutils

# fix for DDE
Preinstall: libpam-runtime libpthread-stubs0-dev util-linux

Runscripts: base-passwd user-setup base-files gawk

#VMinstall: libdevmapper1.02.1
VMinstall: binutils libblkid1 libuuid1 mount libmount1 libsmartcols1 util-linux
%ifarch ppc64le
# for LE switch
VMinstall: kernel-obs-build
%endif

%ifarch armv7l armv7hl
%if "0%{?_obs_feature_exclude_cpu_constraints}"
Constraint: hardware:cpu:flag exclude=true EL0
%endif
%endif

Order: user-setup:base-files

# Essential packages (this should also pull the dependencies)
Support: base-files base-passwd bash bsdutils coreutils dash debianutils
Support: diffutils dpkg e2fsprogs findutils grep gzip hostname libc-bin 
Support: login mount ncurses-base ncurses-bin perl-base sed init-system-helpers 
Support: sysvinit-utils tar util-linux

# for unpacking sources in build script
Support: cpio

# Build-essentials
Required: build-essential
Prefer: build-essential:make make

Prefer: sysvinit-utils

# build script needs fakeroot
Support: fakeroot
# lintian support would be nice, but breaks too much atm
#Support: lintian

# helper tools in the chroot
Support: less kmod net-tools procps psmisc strace vim

# Workaround/Hack, it is declared as dummy package
Prefer: -libfontconfig1-dev
Prefer: -libfreetype6-dev

# everything below same as for Debian:6.0 (apart from the version macros ofc)

# circular dependendencies in openjdk stack
Order: openjdk-6-jre-lib:openjdk-6-jre-headless
Order: openjdk-6-jre-headless:ca-certificates-java

Prefer: cvs libesd0 libfam0 libfam-dev expect
Prefer: -bzr # in favor to brz
Prefer: -libelogind0

Prefer: gawk locales default-jdk sudo golang-pq-dev imagemagick
Prefer: xorg-x11-libs libpng fam mozilla mozilla-nss xorg-x11-Mesa
Prefer: unixODBC libsoup glitz java-1_4_2-sun gnome-panel
Prefer: desktop-data-SuSE gnome2-SuSE mono-nunit gecko-sharp2
Prefer: apache2-prefork openmotif-libs ghostscript-mini gtk-sharp
Prefer: glib-sharp libzypp-zmd-backend mDNSResponder libcom-err2

Prefer: -libgcc-mainline -libstdc++-mainline -gcc-mainline-c++
Prefer: -libgcj-mainline -viewperf -compat -compat-openssl097g
Prefer: -zmd -OpenOffice_org -pam-laus -libgcc-tree-ssa -busybox-links
Prefer: -crossover-office -libgnutls11-dev

# for DDE
Prefer: libavcodec58 libavformat58 libavfilter7 libjpeg-dev
Prefer: dnsmasq-base libncurses-dev node-unicode-14.0.0 fontforge-nox
Prefer: node-debbundle-acorn 

# alternative pkg-config implementation
Prefer: -pkgconf
Prefer: -openrc
Prefer: -file-rc

#emacs
Prefer: sendmail-bin

#aubio
Prefer: libjack-dev

#libde265
Prefer: libsdl1.2-compat-dev

#nlopt
Prefer: gnuplot-qt

#golang-github-hashicorp-go-msgpack
Prefer: golang-gopkg-check.v1-dev

#gst-plugins-bad1.0
Prefer: libmagick++-dev libsdl1.2-compat-shim

#node-hooker
Prefer: uglifyjs

#maxima
Prefer: emacs-nox

#ruby-image-processing
Prefer: ibmagickcore-dev libmagickwand-dev

Prefer: fuse3

#llvm-toolchain-14
Prefer: libcurl4-gnutls-dev

Prefer: devscripts:fakeroot
Prefer: bsdutils:libsystemd0

Conflict: ghostscript-library:ghostscript-mini

#Ignore: sysvinit:initscripts

Ignore: aaa_base:aaa_skel,suse-release,logrotate,ash,mingetty,distribution-release
Ignore: gettext-devel:libgcj,libstdc++-devel
Ignore: pwdutils:openslp
Ignore: pam-modules:resmgr
Ignore: rpm:suse-build-key,build-key
Ignore: bind-utils:bind-libs
Ignore: alsa:dialog,pciutils
Ignore: portmap:syslogd
Ignore: fontconfig:freetype2
Ignore: fontconfig-devel:freetype2-devel
Ignore: xorg-x11-libs:freetype2
Ignore: xorg-x11:x11-tools,resmgr,xkeyboard-config,xorg-x11-Mesa,libusb,freetype2,libjpeg,libpng
Ignore: apache2:logrotate
Ignore: arts:alsa,audiofile,resmgr,libogg,libvorbis
Ignore: kdelibs3:alsa,arts,pcre,OpenEXR,aspell,cups-libs,mDNSResponder,krb5,libjasper
Ignore: kdelibs3-devel:libvorbis-devel
Ignore: kdebase3:kdebase3-ksysguardd,OpenEXR,dbus-1,dbus-1-qt,hal,powersave,openslp,libusb
Ignore: kdebase3-SuSE:release-notes
Ignore: jack:alsa,libsndfile
Ignore: libxml2-devel:readline-devel
Ignore: gnome-vfs2:gnome-mime-data,desktop-file-utils,cdparanoia,dbus-1,dbus-1-glib,krb5,hal,libsmbclient,fam,file_alteration
Ignore: libgda:file_alteration
Ignore: gnutls:lzo,libopencdk
Ignore: gnutls-devel:lzo-devel,libopencdk-devel
Ignore: pango:cairo,glitz,libpixman,libpng
Ignore: pango-devel:cairo-devel
Ignore: cairo-devel:libpixman-devel
Ignore: libgnomeprint:libgnomecups
Ignore: libgnomeprintui:libgnomecups
Ignore: orbit2:libidl
Ignore: orbit2-devel:libidl,libidl-devel,indent
Ignore: qt3:libmng
Ignore: qt-sql:qt_database_plugin
Ignore: gtk2:libpng,libtiff
Ignore: libgnomecanvas-devel:glib-devel
Ignore: libgnomeui:gnome-icon-theme,shared-mime-info
Ignore: scrollkeeper:docbook_4,sgml-skel
Ignore: gnome-desktop:libgnomesu,startup-notification
Ignore: python-devel:python-tk
Ignore: gnome-pilot:gnome-panel
Ignore: gnome-panel:control-center2
Ignore: gnome-menus:kdebase3
Ignore: gnome-main-menu:rug
Ignore: libbonoboui:gnome-desktop
Ignore: postfix:pcre
Ignore: docbook_4:iso_ent,sgml-skel,xmlcharent
Ignore: control-center2:nautilus,evolution-data-server,gnome-menus,gstreamer-plugins,gstreamer,metacity,mozilla-nspr,mozilla,libxklavier,gnome-desktop,startup-notification
Ignore: docbook-xsl-stylesheets:xmlcharent
Ignore: liby2util-devel:libstdc++-devel,openssl-devel
Ignore: yast2:yast2-ncurses,yast2-theme-SuSELinux,perl-Config-Crontab,yast2-xml,SuSEfirewall2
Ignore: yast2-core:netcat,hwinfo,wireless-tools,sysfsutils
Ignore: yast2-core-devel:libxcrypt-devel,hwinfo-devel,blocxx-devel,sysfsutils,libstdc++-devel
Ignore: yast2-packagemanager-devel:rpm-devel,curl-devel,openssl-devel
Ignore: yast2-devtools:perl-XML-Writer,libxslt,pkgconfig
Ignore: yast2-installation:yast2-update,yast2-mouse,yast2-country,yast2-bootloader,yast2-packager,yast2-network,yast2-online-update,yast2-users,release-notes,autoyast2-installation
Ignore: yast2-bootloader:bootloader-theme
Ignore: yast2-packager:yast2-x11
Ignore: yast2-x11:sax2-libsax-perl
Ignore: openslp-devel:openssl-devel
Ignore: java-1_4_2-sun:xorg-x11-libs
Ignore: java-1_4_2-sun-devel:xorg-x11-libs
Ignore: kernel-um:xorg-x11-libs
Ignore: tetex:xorg-x11-libs,expat,fontconfig,freetype2,libjpeg,libpng,ghostscript-x11,xaw3d,gd,dialog,ed
Ignore: yast2-country:yast2-trans-stats
Ignore: susehelp:susehelp_lang,suse_help_viewer
Ignore: mailx:smtp_daemon
Ignore: cron:smtp_daemon
Ignore: hotplug:syslog
Ignore: pcmcia:syslog
Ignore: avalon-logkit:servlet
Ignore: jython:servlet
Ignore: ispell:ispell_dictionary,ispell_english_dictionary
Ignore: aspell:aspel_dictionary,aspell_dictionary
Ignore: smartlink-softmodem:kernel,kernel-nongpl
Ignore: OpenOffice_org-de:myspell-german-dictionary
Ignore: mediawiki:php-session,php-gettext,php-zlib,php-mysql,mod_php_any
Ignore: squirrelmail:mod_php_any,php-session,php-gettext,php-iconv,php-mbstring,php-openssl

Ignore: simias:mono(log4net)
Ignore: zmd:mono(log4net)
Ignore: horde:mod_php_any,php-gettext,php-mcrypt,php-imap,php-pear-log,php-pear,php-session,php
Ignore: xerces-j2:xml-commons-apis,xml-commons-resolver
Ignore: xdg-menu:desktop-data
Ignore: nessus-libraries:nessus-core
Ignore: evolution:yelp
Ignore: mono-tools:mono(gconf-sharp),mono(glade-sharp),mono(gnome-sharp),mono(gtkhtml-sharp),mono(atk-sharp),mono(gdk-sharp),mono(glib-sharp),mono(gtk-sharp),mono(pango-sharp)
Ignore: gecko-sharp2:mono(glib-sharp),mono(gtk-sharp)
Ignore: vcdimager:libcdio.so.6,libcdio.so.6(CDIO_6),libiso9660.so.4,libiso9660.so.4(ISO9660_4)
Ignore: libcdio:libcddb.so.2
Ignore: gnome-libs:libgnomeui
Ignore: nautilus:gnome-themes
Ignore: gnome-panel:gnome-themes
Ignore: gnome-panel:tomboy
Ignore: gcc-11:doxygen

Substitute: utempter

Substitute: java2-devel-packages java-1_4_2-sun-devel

Substitute: yast2-devel-packages docbook-xsl-stylesheets doxygen libxslt perl-XML-Writer popt-devel sgml-skel update-desktop-files yast2 yast2-devtools yast2-packagemanager-devel yast2-perl-bindings yast2-testsuite

#
# SUSE compat mappings
#
Substitute: gcc-c++ gcc
Substitute: libsigc++2-devel libsigc++-2.0-dev
Substitute: glibc-devel-32bit
Substitute: pkgconfig pkg-config

%ifarch x86_64
Substitute: kernel-binary-packages kernel-default kernel-smp kernel-xen kernel-kdump
%endif

%define deepin 1
%define deepin_version 23

Macros:
%deepin 1
%deepin_version 23
myml commented 1 year ago

客户端 osc 使用

osc 是 obs 的客户端,对接web接口,可以方便管理

配置

[general]
apiurl = https://x.x.x.x # 默认服务器地址,使用 -A 可指定其他服务器

[https://x.x.x.x]
user=Admin # 用户名
pass=opensuse # 密码
sslcertck=0 # 不检查https证书

项目列表:osc list 拉取项目:osc checkout $project_name 创建包:osc mkpac $package_name 添加文件:osc add $file 提交文件:osc commit 从上游更新:osc up 调试构建环境:osc shell 提前使用 stty intr ^] 更改中断,避免 ctrl+c 导致 kvm 退出

更改仓库下载链接:osc api -e /configuration 修改 download_url 字段

myml commented 1 year ago

管理端 bs_admin 使用

触发项目检查: /usr/lib/obs/server/bs_admin --check-project $project x86_64 列出被拉黑的构建节点:/usr/lib/obs/server/bs_admin --list-badhosts 取消构建节点拉黑状态:/usr/lib/obs/server/bs_admin --drop-badhosts

myml commented 7 months ago

常用配置项 /etc/sysconfig/obs-server

OBS_WORKER_JOBS 构建并发数,即 make -j后面的数字,默认是1,建议设置大些,加快单个任务的构建 OBS_WORKER_INSTANCES 一个节点的构建队列数,默认是节点的CPU数,建议结合 JOBS 调整少一些,避免节点卡死 OBS_INSTANCE_MEMORY 构建虚拟机的内存,默认512M太小了,构建大型项目会很慢,建议调整大一些 OBS_VM_DISK_AUTOSETUP_ROOT_FILESIZE 构建虚拟机的磁盘大小,默认4G够用了,构建大型项目可以调整大一些

shafeipaozi commented 3 months ago

楼主您好,我也在使用OBS,遇到了一些问题,想跟您请教一下 1.添加了repositories使用DoD模式,但是并没有触发下载,在对应工程架构下不存在:full文件夹,或者文件夹内无依赖软件包 2.我当前的机器是x86,我想构建aarch64和riscv等等其他架构的软件包,我该如何给工程添加一个新的指定架构的worker呢

myml commented 3 months ago

@shafeipaozi 项目config添加 Type: dsc Repotype: debian 查看DoD仓库的状态,是不是绿色图标

可以在其他架构的机器上安装 obs-worker 这个包,修改 OBS_REPO_SERVERS 配置,让worker连接到你的obs服务

shafeipaozi commented 3 months ago

@myml 感谢回复,我去尝试一下! 1.我在添加您说的这两行前也是绿色卡车,添加后也为绿色卡车(我的系统是rpm包),我就换成了rpm都 我尝试添加了之后软件包从unresolved状态转变成了excluded状态(我可以确信我打开了target,并且spec文件没有exclude声明),然后我深入文件/srv/obs/build/project/repo/x86_64/:full/doddata.cookie 在这个:full目录下只有一个doddata.cookie,没有预期的相关依赖依赖。 2.关于这部分,我的理解是,在aarch机器上面只需要install obs-worker软件包,不要安装一个OBS的ISO镜像了,然后您说的这个 OBS_REPO_SERVERS 配置,是在服务本机配置还是新的aarch机器上面配置呢,这个配置项的大概url您还记得吗?

myml commented 2 months ago

@myml 感谢回复,我去尝试一下! 1.我在添加您说的这两行前也是绿色卡车,添加后也为绿色卡车(我的系统是rpm包),我就换成了rpm都 我尝试添加了之后软件包从unresolved状态转变成了excluded状态(我可以确信我打开了target,并且spec文件

仓库是绿色卡车状态,一般是没问题的,你可以先正常去使用这个仓库,DoD是按需加载,并不会下载整个仓库,在构建软件包的时候再看日志有没有相关报错之类的。

2.关于这部分,我的理解是,在aarch机器上面只需要install obs-worker软件包,不要安装一个OBS的ISO镜像了,然后您说的这个 OBS_REPO_SERVERS 配置,是在服务本机配置还是新的aarch机器上面配置呢,这个配置项的大概url您还记得吗?

OBS_REPO_SERVERS需要配置到arm的机器上,obs的worker通过这个配置连到obs的主服务。这个字段可以写ip或者域名

shafeipaozi commented 2 months ago

@myml 感谢回复,我去尝试一下! 1.我在添加您说的这两行前也是绿色卡车,添加后也为绿色卡车(我的系统是rpm包),我就换成了rpm都 我尝试添加了之后软件包从unresolved状态转变成了excluded状态(我可以确信我打开了target,并且spec文件

仓库是绿色卡车状态,一般是没问题的,你可以先正常去使用这个仓库,DoD是按需加载,并不会下载整个仓库,在构建软件包的时候再看日志有没有相关报错之类的。

2.关于这部分,我的理解是,在aarch机器上面只需要install obs-worker软件包,不要安装一个OBS的ISO镜像了,然后您说的这个 OBS_REPO_SERVERS 配置,是在服务本机配置还是新的aarch机器上面配置呢,这个配置项的大概url您还记得吗?

OBS_REPO_SERVERS需要配置到arm的机器上,obs的worker通过这个配置连到obs的主服务。这个字段可以写ip或者域名

1.DoD是按需下载,我简单添加了一个软件包(他的buildrequire只有三个),我的绿色卡车就会报unresolved说缺少这三个依赖,我去找到工程架构下面的:full目录也确实没有这些依赖软件包,换句话说,DoD并没有启动进行下载(DoD的链接可以点击手动打开)(我比较怀疑是project config的问题,这里我一直没能读懂) 2,arm的机器还有等一阵子才能到手,我在官网找到了OBS_REPO_SERVERS这个说明,OBS_REPO_SERVERS="",跟一下arm机器的ip就可以吧

shafeipaozi commented 2 months ago

楼主,我还遇到了一个新的问题,在我确认repo是绿色卡车后,开始构建软件包,但是在preinstall之后,会出现 [ 17s] initializing rpm db... [ 17s] chroot: failed to run command '/usr/bin/rpmdb': No such file or directory 会出现这样的错误,请问我该如何解决这个问题呢,我猜测可能是我的project config内存在对rpm db的影响,但是具体是由于哪里引起的或者他缺少哪些,我是很困惑的!

myml commented 2 months ago

楼主,我还遇到了一个新的问题,在我确认repo是绿色卡车后,开始构建软件包,但是在preinstall之后,会出现 [ 17s] initializing rpm db... [ 17s] chroot: failed to run command '/usr/bin/rpmdb': No such file or directory 会出现这样的错误,请问我该如何解决这个问题呢,我猜测可能是我的project config内存在对rpm db的影响,但是具体是由于哪里引起的或者他缺少哪些,我是很困惑的!

看上去是少安装了一些软件导致的,你可以搜索rpmdb是在哪个软件包里面,安装一下就可以

shafeipaozi commented 2 months ago

楼主那个问题我解决了,是在project config中需要解决这个问题 我还有个疑问,就是我的对应架构的服务突然会挂掉,需要我使用restart obsscheduler来不断的重启服务(在下面的status monitor中目标架构是x的状态),我查看了log日志并没能找到原因,楼主你有遇到类似问题吗?

myml commented 2 months ago

楼主那个问题我解决了,是在project config中需要解决这个问题 我还有个疑问,就是我的对应架构的服务突然会挂掉,需要我使用restart obsscheduler来不断的重启服务(在下面的status monitor中目标架构是x的状态),我查看了log日志并没能找到原因,楼主你有遇到类似问题吗?

这个没遇到过,还是要从日志中找问题

shafeipaozi commented 1 month ago

楼主那个问题我解决了,是在project config中需要解决这个问题 我还有个疑问,就是我的对应架构的服务突然会挂掉,需要我使用restart obsscheduler来不断的重启服务(在下面的status monitor中目标架构是x的状态),我查看了log日志并没能找到原因,楼主你有遇到类似问题吗?

这个没遇到过,还是要从日志中找问题

楼主,上次的问题解决了,但是我最近有新的想法,想将suse的一款icecream分布式编译结合给obs来解决大型软件包的构建问题,您有做过类似工作或者想法吗,想和您交流一下!

myml commented 1 month ago

楼主那个问题我解决了,是在project config中需要解决这个问题 我还有个疑问,就是我的对应架构的服务突然会挂掉,需要我使用restart obsscheduler来不断的重启服务(在下面的status monitor中目标架构是x的状态),我查看了log日志并没能找到原因,楼主你有遇到类似问题吗?

这个没遇到过,还是要从日志中找问题

楼主,上次的问题解决了,但是我最近有新的想法,想将suse的一款icecream分布式编译结合给obs来解决大型软件包的构建问题,您有做过类似工作或者想法吗,想和您交流一下!

这个我没接触过