Closed wzgrx closed 1 year ago
If the folder is on Google Drive, the original copying method is very complex and troublesome You can refer to the following Collab code
https://colab.research.google.com/github/WSH032/lora-scripts/blob/main/Colab_Lora_train.ipynb#scrollTo=j6nkqp9Fb7Dg
/content/lora-scripts/train/
use_data_dir_self = False #@param {type:"boolean"} copy_reg = False #@param {type:"boolean"}
train_data_dir_self = "/content/drive/MyDrive/Lora/input/" #@param {type:'string'} reg_data_dir_self = "/content/drive/MyDrive/Lora/reg/" #@param {type:'string'}
if use_data_dir_self: print(f"你使用的是自定义路径") else: train_data_dir_self = "/content/drive/MyDrive/Lora/input/" reg_data_dir_self = "/content/drive/MyDrive/Lora/reg/" print(f"你使用的是默认路径") print(f"训练集地址为:{train_data_dir_self}")
!mkdir -p /content/lora-scripts/train/ #防止首次运行报错 !rm -r /content/lora-scripts/train/
print("拷贝训练集中") !mkdir -p {train_data_dir} !cp -r {train_data_dir_self}/* {train_data_dir} !echo "copy训练材料完成."
if copy_reg:
print(f"正则化集地址为:{reg_data_dir_self}") print("拷贝正则化集中") !mkdir -p {reg_data_dir} !cp -r {reg_data_dir_self}/* {reg_data_dir} !echo "copy正则化图片完成." else: print("不拷贝正则化集")
%cd /content/lora-scripts
Why we need to copy the train data dir? I think you can just mount the drive and copy the drive folder path.
If the folder is on Google Drive, the original copying method is very complex and troublesome You can refer to the following Collab code
https://colab.research.google.com/github/WSH032/lora-scripts/blob/main/Colab_Lora_train.ipynb#scrollTo=j6nkqp9Fb7Dg
@title ####拷贝材料(支持重复训练时选择新的路径)
@markdown 默认给的是教程(三)中的谷歌硬盘目录,你也可以自定义训练集和正则化集路径(只要你知道你在做什么),请选择repeat_concept的父目录
@markdown 为了稳定性和性能,notebook会把材料拷贝到colab的
/content/lora-scripts/train/
中进行操作@markdown 是否使用自定义路径,是否拷贝正则化图片
use_data_dir_self = False #@param {type:"boolean"} copy_reg = False #@param {type:"boolean"}
@markdown 自定义训练集路径,正则化集路径(仅在勾选后有效)(不要使用带空格、中文的路径)
train_data_dir_self = "/content/drive/MyDrive/Lora/input/" #@param {type:'string'} reg_data_dir_self = "/content/drive/MyDrive/Lora/reg/" #@param {type:'string'}
if use_data_dir_self: print(f"你使用的是自定义路径") else: train_data_dir_self = "/content/drive/MyDrive/Lora/input/" reg_data_dir_self = "/content/drive/MyDrive/Lora/reg/" print(f"你使用的是默认路径") print(f"训练集地址为:{train_data_dir_self}")
@markdown 拷贝时间由训练材料大小决定
@markdown 出现这样的输出就是正确放置了文件且完成了拷贝
@markdown![copy.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAARcAAABtCAIAAAAEf7KQAAAV4klEQVR4Ae2dSaglxRKG7eatnKA3jiD6BHXhgNrgiL5nuxFxAhWcUXBAsNtZQRERwaEVp40KKuIADojaihsVEXFARRwWKjhsnBeCtm77fd3/IzrMzMpb51Sdun3qRi0OWZERkZlR8WdkZmXlWbZhw4at4goLhAU6WGB5B9kQDQuEBTZaIFAUfhAW6GqBQFFXC4Z8WCBQFD4QFuhqgUBRVwuGfFggUBQ+EBboaoFAUVcLhnxYIFAUPhAW6GqBQFFXC4Z8WCBQFD4QFuhqgUBRVwuGfFggUBQ+EBboaoFAUVcLhnxYIFAUPhAW6GqBQFFXC4Z8WCBQFD4QFuhqgUBRVwuGfFggUBQ+EBboaoFAUVcLhnxYIFAUPhAW6GqBQFFXC4Z8WCBQFD4QFuhqgUBRVwuGfFggUBQ+EBboaoFAUVcLhnxYIFAUPhAW6GqBQFFXC4Z8WCBQ1LMPfPTRRwcffPCrr77q9XJ7zDHH/Pjjj56YpK+44oqLLrqoyEPWjTfe+OeffyYi3JLFldOVddZZZxWljJ/aUmidx5gj0WSBQFHZMvg9l+UVsXH//ffnbvruu+8eddRRxx9/vMmSeP3116+66qpddtnFE/P0vvvu28Szww47bLfddrkIlGOPPbZIh3jIIYc0SUlk5cqVFHrJJZdMCiT4aTvXV199BYwnFW+q8JzSA0XlBwcMcH28xPyDCAM8iBU4zbPPPiuxxE1h/uKLL5AiFxD+e9PF7aeffnrZZZfpVr8gMCl4jz32SCiVW1D91ltvGQO1ItZR4qOPPkqW0fMEPCeeeKKvyb333vvee+8dcMABnkg6r6HXBjgBNs3fe++9aeAJJ5xQL9fLji/9r0mbdN99933wwQcVKSy7Zs2aCsO8ZF2z6QIYSXf+66+/HnnkkcVWvP3223TtdPDkgkMuEjjuN998s3r16qLIdER8l+C27bbbCnug6PDDD6c4aktgwb/vueeeoma6BmLXyy+/XMwVETy8+OKL7StMew888ECCsBpe0TzWrIlRhCHef//9sZrDt4vB1VNPPeUppNUHJ0Td4sr46C233MItyLnpppseeughHIue/uSTTy6K4K8XX3zx77//Ti7wI44RGUjj6H/99ReCpHffffedd975pZdeImvFihXSSTXgx3Gl9o033rj00ktJQydUPvzwwxZCxaBfivvkk08UKj09SX/22WfXXXedJ/p6ejppVVhE0ieddFITgBPBMd1OgyLaT7QpBhxGAmOyTvu24LU333zzmWeeSShYtWoVffndd9+NP22zzTYff/zx008/LVUPPPCAApRuwRi5SuP9gCePAIDz7LPPvvzyy5OsI444AnQ9//zzv/zyC4I2oUInFzrzISJ0xoEgvPKYqOEFF1ygKvnfXXfddd26dVYKWeih11iCmPFmUXpKFOWKRkPRoMjiwJNPPuldp6mZCg6MsogVRDAizH82XcwuCBoAALVEJ0Y+RQ1AhbkT16mnnpoUB06+//77RArIEZ0IUwzkLBq0jAMbB5r/XPxIlLe/ZXBLu2g7Il9++eU+++zTXnZMnD2vLnz77bfW7/ZuJsYVTPE1CU7mvupflUVaRcMDhV8lSON8yjJVDG/wA2NAFp8ABjSEXpmRWOLTlUYBFQILkYEgIDdFM4M0HL0ipSwGWkAFTsIOlTF+jRJ32203oyhBBKCGIJxBHfUkzQURWbUo4Z/F7Y477kgDuVBOPRlSDlb0LJrTRWfPKLKqsAjBZbfdExqaM+vAXd555x3GElCkFtenm2fdTFkMpQQk3JqBEJywkUVPT98pIGlsQ+e9du1aYAMn0QO1HTtpqkQQQ5sqxiowE32Nr+oWoJKELJjvvPPOxx9/3DoCqsdorSmCCXus/hk/RaMEaxSLs76DDoX+iFusQRrvZ6RnPRSU/O3WDz/8wJoKWboQVP8CfiiLgMnw8sEHH1RcKpY+YuKsUMSsCRRh8b5sR3w477zzzMvXr19Pr49y+kKiH+iSs/JoGVZBUR8Jw5577im35gHz7Fku04PnlqCBI8IjSvuwU2wUTsm0nnk5Pk3Dcc1rr72WmsjtWEpmDGaO6LttBFlLUMiiFccddxzdPEWIfv755xeLo4EoBGPEIkBofo9mJmNFEZQTLelQuIANsQtx0piFekLZlLPx580330ysAcbohsjiFylKgQHb8hSoCcu2yaJ/sQJjJfY8L/LLdzwYZtv4EKbvaD6eE2GEhyc9PD8es9L093STcjtRSEOBLlz5SbY8A8wowSTkiSeeAJlgiVBQqSS+gmsCPANnwgwdpyQQgRZCXN5kGAgRLJ6raC/OqAwR4C3irbfeSoJKolCh0jNbWoGOyAOFTsTiD61Ww6GDLr+yZ7JK0CJvtyS3zS22Vaj/6aefMGYbkVHy9IwibARyEktBAVEJcaJbXJDn1CSy/fbbe9ckDaXIjGv+8ccflgUnmunyAclpp51mdJ8glwU3nNWj0TMoDcDyZXGy8HV+eb0jNioAJ5du+UU/HQTRMpkOgTcg5NtlIiSoNsXBw1CKWxQCORIoscUGbgFn06shNBAAm/Qjm1xYwFdbucTPF154gT6o2DskGkZ82/OI7tBDD6UbtotbLpsnTG1Hnh/DbuutpYf39LgCDkquBmaik4ZijusLBS3Lly/3rnPYYYcxDIMndxGIf//9N30t8+aWiwS+LM1VmCrQiQAVZS1btoyxkN1Sf/TnL2dYLq9ACFXPPfecxlRSCzgZ0RH2uTXjMyesIB8rAQxvCqnSL6M7qyQU7KagR9rDj6DHnI1FcHZ1mCAMSi+d355R5A2noASEAJKnT5HGxdHGtNseLZ0uoxHoymJFAU9CM7+kYYaugkxKWcmymx7//vvvn9cK0OKsDCMRxIc0TMJFCB05s1E0KZJDU1uU8JrFhlj0BcQQ4gZTEUTQfMopp6iq3333HRThCp4m/4ZHRpBOScHMEBf8tO+wQK+NkK3yTQlQRNjB5lSbOmtwCzONBY2scWs8SeVtSA+nFjCadI6J3v+ITtbBmkw3ubpDSAq1rnD66afrlo7WVhqUsF05zLYtC2aGapqxkM7fXUKkhublUq5f/IYtNmSxn7pp84Hx41ssUrNabZMiKIxziEUJHsAMHnb11VcDhrxccq+//npTW0wglQvmnACsKRZp9OitlIizfQGKSlHHQWznFtjwCDAj9STNQJ0+AgbaTmwESwRbdQqJwnHfzgpFuCZXv7bjqTc9+ErW1ltvXZyxqG7sfGuaYW901ZUr8Xj6bBLiBxJFbdB5e4MDyYcEIY3K0MBiGuK8JJUStJ177rnFMacY9IubsoqtNF2Dz5o6LbTTKWgeJT3UhDkS6yJeLbXVK2BMtN9++6ldZmcaBbNMQRZtB0isLtKJSInX79WOM71hwovJKz0cv0U5ss4444xi1vBEltqpDxcDPBYVkgpo2sBvQve35L7yyiue4tPPPPMMi4GeojRlIeizoFCHiioEEaHCubYPP/zwkUceyemi3HDDDTAUc5PKU5//brp8xYqCngjz7bff7in1tjTZxGsYX3oZTZqoe+BJc23aRrcmF2Q+QAjquCKXqw1KWGBLtsCUIzphaUtuWNQtLDCYBSZGUY8LBoM1MgoKC8zUAhOP6GZam1AeFphHC8zwfdE8mmOAOrPAzTcTeufji2O9mMVirX1BZ4Gu/r6F1Wr79htO+PXilalpcpHlCyqmKdeKLjJ4IpWPM0+8QQJF3hqb0/ho4ov4KFs5c0/1zuc928TBBgjZrHqrrfbaa698mVsbAmE2Tt4I2wq7EX2CJXjtVGCrASs6bMO1jQtKsEudzRn5lnDqSUNY9U60+dtKmlqx7TDOPDETTTwvMslxJ9gEkOwD0CugBT/aw7P91jX8lY3PetlStxh7UnH3OmyaNPBKh32GOVqa+KEnezgqnGQRzfj6kG1QCVvyiolcltcTu3kR7MAuEN490InQX3DmyV133TVdk73aRU8Hivp8BAAm2exn2nFEtiYBQqP4BMGKjwt4p+SJ7dNs52GbhXZvEHzYocPV43tPbRfyvUNeN4Z5S/bMk0BR7g89UAhBoILtP14XPTdbHJJxlBgIJiRsT6eXIo2D8hEhuUVggE96dzYNMDuis4eZDyWSohOFE92ikM1yfqhZFF/KZ54Eioou8X8i8w25Zo2pIY9XAm0GckgTiJjV8G0SWMq3OIE6vvbjC8XiSIlcBlrsCsfX2TurTYNALtm8l9eR3Sdcns42JdtwBN3GZoy4sANYZabn+X2aDUpL+cyTQJF3hjRN/0q/vuBcKBWb8F6BqLgMAMDY0socpgghymH94JxzziGBr7OicMcdd4DeHIp5jQwkeVaRYjvoirkTEdkUT7vUxYzjzJN5WqOjuyUyaO2LIZN/cuoplUVaWfBA4VcJ0raeZqq0gGYMJmvK/SycGbx98q2yTKHxT5rAn5h1XHnllUVBdkmDiiYIIcIOa4UdmsBm8Ndee01fnhe1bQlE9v7SZC4qQyAdx5knc4Mi/J6BCpGBHnexTi/h6yCK9qvJxYnKRM762GOPsayH5lyKIRYTngqEJKIegc+oOL+FoRdVylexc+XtKdbj0HHQi3FL30GaDkgjXnUo/OblspnV9zsICvOaHI7mzJO5GdGxCX+i00uYi2vMkJxeAg55hDxLcvFR5s2k9VD1gNu7V52TcusM5FLuTjvtxGBJFTB+blnQY8GgDiFmRLg1R6vquwmmTww+MRSOziIydEaJptMn+PQIiCbzIhjgZ1kimc4xVuRFmcSJIbwmYi0RrFIKnwkDpITfF0T1aAgMCPI6C+BhZCzDZ39QRnPmyXygCIsv7ukl3jMqaVvmxk3boAiXsu8OTS2jSjyPz3U+//xzIxYTfBTknZj3RUixEoiLy+9BI9OqRFbGTL5lhIcxIb8VSEgPa+hNX2QlBTXd0nkxiCWXEfI4zjyZGxRh8aanMtPTS5oKrdDxXTCvd6D1YJIooY2CAZ9/g6UERQBG396aVLIMCCbtaCTjyROgiGs6JCAYZ57kJp2PeREd5KKcXpLbq04BM1w6GIRBf7L2wPynLv7zzz8zJCOS1NnyXFsdYXKikJLzGIXqYc98C5IxVBJqWtPQF5wzzDPxpXPmydygiLG4nUPCc8JX6E3xBi6yGAJpaqEZBRToepwmpSy/5gaDXnQWTy8xb6gkWKhNcumtmXIwAQBOwAa31rpfm5ViwleTg1IKQQC/TIrTLWXZmgeDyQS9iQijKW+fJLd+q00MdR7LXTpnnszHiI4Ho3cgNovgdYcoltXX6SXAgAm0TruXQ/hZuJVivpLMyFlzY6em9oZRQ2ZKrFYDeFFMqk3Cv1eBH1VssfFI2/je9J9vTk0tSFbaDnvQrUHaONsnFOXM7Lngkj3zZG5QxDPj+TU9wkrWpKeXEMSK55PkTpNTCALJ2jTnA2sTJ5GQOY/+/SFfOiMX9HqFoIUAm2z3ZNXOB6sF35yi1q8uMNyid5hixKjKE8a9bJx5svl5je8oCWsRn7Xj01wMYKY7vcRULZion+lh4rBdeOGF+XkjUKDnlTTBPKGP9nO6p/izR1B+2223VYoAk+j04qTREGeeJDbJb+Nb180dSqTCAtNZYD5WF6ZrW0iFBYaxQKBoGDtHKWO2QKBozE832jaMBQJFw9g5ShmzBQJFY3660bZhLDDHKOL/i/x+k2HsNUApY23XAKZbrCLmFUW8B/ztt9+m2BCwWIZuWW7eLt7G8jIXeksNwTa8BeYVRV9//fWqVauGt9esS/TtItLyfQ7bF/hD4lmXO7x+9iLZJsPhS++3xLlEEd0zmy9nHYhwYjYp92juBRUm7aKBfCbE17XFL2F7rNiiqGJv4YoVKyYtekEbTqqwF/552kdnDea4j+k+jzENbRIcs9jmS7s2qsSzoMJh2tW+wjPlVB8xaREL2nBShb3wz18sosPm87Wmo9t6MQpKmIewgbovbW0UDtOuHls0vKreH0pfTZg/FNFh2x8kmhU0hWCczZV8pqbxt7L0XQBSIsJpuToMiCy8mTRfQLD/mvPZEEwO5SiW1UWhWlFslzUwT1jNqSFpMRhRnxjx6+uPF9IWKLoSQ+VFGMXUStCKg6FiDZiL1TCRxLDUBxF+lTBxSlnwoVRqaK2YYSLfoLolU4obk9kQfdBBB+n/HtmDzMEdtmkaIlm61fZk+1tI9i/DyUZm2ota9n1bVpEis6AKKVQZj+2Dnk6h1BbbpSzKom4qURR+K+1SLu3igo0KH3300bIAFKstCs1QELX5PfmVQZD1+80hmhKymixfqYZaUWyXbCj9eijUWfz85o9JWZUamuxME3M2Lyp22F2OB9IHM3xTxBkGdvZIpdOiLDt5VJ/ZcqwpPaU+rWUeNalClVVsV1M1KI5COcxI6yt8ccTxOlDs2CME+UrPziexNRL/dRAV5vghFbHxM/fVq5uKS+j+E68my5tIsRqWW0xgQ1WGGhLK7MymInOR6GtYZOidOE8jOrwnnxFB5JtQWwbApTjBQ3bUf5n4dQjSdIHQZUe63okMqrI0zNPYhjRHjkCfTqGkiu2qVGzBdiHL93xFZ7LRFAvodj5WpSyywCpf78Kvw+iMuWJ542mqhjHkCf9Q9Enigu/KmmqYK58RZZ5iUbHD5ln2cjxQe/taH99epM5ZbFddpP2xR14PXTuvntatW4d3YjcimM+tpDfFqtV4M5/rwqYjl+uWr2hrn0WJjOLa8Bdr2EawF565iUU8szwQYQLifi/HA7WxJmXx9zvJwM9PtdsoSXia2pWw+Vs+1aYmHMdjxDbH+hCFqHmXM8cV58He2rVrKbpieatYxwTnnyxfvlwRqY2qpIZtRHrhmRsUNXXYPEsm33bQD0ZhhYeRG3RlVY4HqlgQWQMM/sfeNjGzfEdZhhzKqijxWU0Km9rlZZP01O1iNGvY45QVRnTr16+3piWl2C1ttPZCBIoadKkaRcub7BQJU0gg4tn5M5uabNhUwylKn1JkpmsXfSmvLGGpCBaObH3JVpDyLC06QYdH/GJObiWoBSjYkmMbjE6WlZVoSG6bFNbb5Rul2lr9UehzjW7lij+puReBU8wmq0rmv14qt4bPRaHEpVl1yEW8Acm1RVRkEeRwCKotWVNotTJZ3zRfh7w4k51dYj7OXVBfWJwuT9l5bBliY23X1NZVbG+/YDh1Qf0KzgeK+m1zaNsyLQCEdLxecsTflllbX6tAkbdGpMMC01hgblYXpmlcyIQFBrFAoGgQM0cho7ZAoGjUjzcaN4gFAkWDmDkKGbUFAkWjfrzRuEEsECgaxMxRyKgtECga9eONxg1igUDRIGaOQkZtgUDRqB9vNG4QCwSKBjFzFDJqCwSKRv14o3GDWCBQNIiZo5BRWyBQNOrHG40bxAKBokHMHIWM2gKBolE/3mjcIBYIFA1i5ihk1BYIFI368UbjBrFAoGgQM0cho7ZAoGjUjzcaN4gFAkWDmDkKGbUF/gf12aJ6ntcPjAAAAABJRU5ErkJggg==)
删除之前的训练材料
!mkdir -p /content/lora-scripts/train/ #防止首次运行报错 !rm -r /content/lora-scripts/train/
从谷歌硬盘中拷贝你之前上传的训练材料
print("拷贝训练集中") !mkdir -p {train_data_dir} !cp -r {train_data_dir_self}/* {train_data_dir} !echo "copy训练材料完成."
if copy_reg:
拷贝正则化图片
print(f"正则化集地址为:{reg_data_dir_self}") print("拷贝正则化集中") !mkdir -p {reg_data_dir} !cp -r {reg_data_dir_self}/* {reg_data_dir} !echo "copy正则化图片完成." else: print("不拷贝正则化集")
%cd /content/lora-scripts