Closed zhf-0 closed 4 years ago
history_xs is the history of hyperparameters being optimized in LBFGS by scikit-optimize, typically it shouldn't be empty if LBFGS iteration works correctly.
Can you uncomment line 221 of lcm.py
and see if there is any x2 and neg_log_marginal_likelihood printed?
If no, check if your scikit-optimize is installed correctly. Are you using mac or linux?
If yes (which is unlikely), you can try to change if (neg_log_marginal_likelihood < min(history_fs)): history_xs.append(x2) history_fs.append(neg_log_marginal_likelihood)
to history_xs.append(x2) history_fs.append(neg_log_marginal_likelihood)
to make sure history_xs is non-empty.
Some further thought:
On Fri, Oct 16, 2020 at 4:56 AM zhf-0 notifications@github.com wrote:
hello developers:
Firstly, I want to thank you for offering such a good tool for researching and studying. I have been using it for a while to tackle my own problem and it works great. But recently, a weird error occurs occasionally. When I run the following command
python MLA_loaddata.py -nodes 1 -cores 4 -nrun 60 ...
the program did work in the past, but now it fails with the same code. More importantly, when I change the option to -nrum 40, it still fails, but when I change it to -nrum 20, the program works properly!
Everytimes the program fails, The error message is the same:
Traceback (most recent call last): File "MLA_loaddata.py", line 204, in
main() File "MLA_loaddata.py", line 143, in main (data, model,stats) = gt.MLA(NS=NS, NI=NI, Igiven =giventask, NS1 = max(NS//2,1)) File "/home/repository/GPTune/GPTune/gptune.py", line 182, in MLA modelers[o].train(data = tmpdata, kwargs) File "/home/repository/GPTune/GPTune/model.py", line 132, in train self.train_mpi(data, i_am_manager = True, restart_iters=list(range(kwargs['model_restarts'])), kwargs) File "/home/repository/GPTune/GPTune/model.py", line 169, in train_mpi res = list(executor.map(fun, restart_iters, timeout=None, chunksize=1)) File "/usr/lib/python3.7/concurrent/futures/_base.py", line 598, in result_iterator yield fs.pop().result() File "/usr/lib/python3.7/concurrent/futures/_base.py", line 435, in result return self.get_result() File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in get_result raise self._exception File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/home/repository/GPTune/GPTune/model.py", line 168, in fun return kern.train_kernel(X = data.P, Y = data.O, computer = self.computer, kwargs = kwargs) File "/home/repository/GPTune/GPTune/lcm.py", line 271, in train_kernel self.set_param_array(xopt) File "/home/repository/GPTune/GPTune/lcm.py", line 86, in set_param_array self.theta[i] = x[cpt] TypeError: 'NoneType' object is not subscriptable After some basic debug, I find out that when the program fails, in lcm.py 266 the variables
history_xs = [None] xopt = None
which causing the error. Keep backtracking, in the definition of
def fun(x, *args): ... if (neg_log_marginal_likelihood < min(history_fs)): history_xs.append(x2) history_fs.append(neg_log_marginal_likelihood) ...
I suspect that the program jump out of the if condition, so history_xs = [None] . But I don't konw how to continue to debug, and come here for help. If needed, I can send all files to you for inspecting. Thank you very much.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/gptune/GPTune/issues/1, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABT6SCQUB62ABR767KS3WCDSLAYFPANCNFSM4STHWQWA .
-- Yang Liu Research Scientist Lawrence Berkeley National Laboratory
Thank you for your reply. Following your suggestions, I did some numerical experiments.
Can you uncomment line 221 of lcm.py # print(x2,neg_log_marginal_likelihood) and see if there is any x2 and neg_log_marginal_likelihood printed?
After I uncomment the line, there are contents about x2 and neg_log_marginal_likelihood displayed on the screen. When the program fails, x2 and neg_log_marginal_likelihood are filled with nan
If yes (which is unlikely), you can try to change if (neg_log_marginal_likelihood < min(history_fs)): history_xs.append(x2) history_fs.append(neg_log_marginal_likelihood) to history_xs.append(x2) history_fs.append(neg_log_marginal_likelihood)
I use ubuntu 18.04. At first, history_xs
is non-empty with float numbers. If program fails, finally it become a list filled with nan
.
do you recall which commit was working properly? There aren't many commits since the release of this repository.
I am using commit 6a161aee15319d6ecbfb26ce9132ed97fbd65724
all the time. when nrun = 20
, it works fine; when nrun = 40,60
, it fails. So, I do not think this error has anything to do with the version of GPtune.
how many tuning parameters do you have for your application? I think if history_xs is empty, it could mean that the an ill-posed problem is given to LBFGS
This is a quite simply problem. After running
sol = scipy.optimize.minimize(...)
I print sol.jac
and find out, when the program fails, jac = [nan,nan...]
. So, maybe the error is about the optimizer LBFGS, and the jacobi matrix is wrong. Do you have any clue about the jacobi matrix?
Thanks for the information. It looks like the optimizer indeed failed. How many categorical values of the tuning parameter do you have? It might be that gptune option you use (options['model_class'] = 'Model_LCM') cannot handle categorical variables well, or is buggy in terms of computing the jacobian.
Alternatively, can you also try set options['model_class'] = 'Model_GPy_LCM' to see if it works?
On Sat, Oct 17, 2020 at 9:49 AM zhf-0 notifications@github.com wrote:
Thank you for your reply. Following your suggestions, I did some numerical experiments.
Can you uncomment line 221 of lcm.py
print(x2,neg_log_marginal_likelihood)
and see if there is any x2 and neg_log_marginal_likelihood printed?
After I uncomment the line, there are contents about x2 and neg_log_marginal_likelihood displayed on the screen. When the program fails, x2 and neg_log_marginal_likelihood are filled with nan
If yes (which is unlikely), you can try to change if (neg_log_marginal_likelihood < min(history_fs)): history_xs.append(x2) history_fs.append(neg_log_marginal_likelihood) to history_xs.append(x2) history_fs.append(neg_log_marginal_likelihood)
I use ubuntu 18.04. At first, history_xs is non-empty with float numbers. If program fails, finally it become a list filled with nan.
do you recall which commit was working properly? There aren't many commits since the release of this repository.
I am using commit 6a161aee15319d6ecbfb26ce9132ed97fbd65724 all the time. when nrun = 20, it works fine; when nrun = 40,60, it fails. So, I do not think this error has anything to do with the version of GPtune.
how many tuning parameters do you have for your application? I think if history_xs is empty, it could mean that the an ill-posed problem is given to LBFGS
- the number of task parameter is 1, which is a categorial parameter and represents the type of matrix to be solved iterativelly;
- the number of input parameter is also 1, which is a real number and represents a parameter in AMG(Algebraic MutiGrid method);
- the number of output parameter is 1, which represents the number of iteration.
This is a quite simply problem. After running
sol = scipy.optimize.minimize(...)
I print sol.jac and find out, when the program fails, jac = [nan,nan...]. So, maybe the error is about the optimizer LBFGS, and the jacobi matrix is wrong. Do you have any clue about the jacobi matrix?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/gptune/GPTune/issues/1#issuecomment-711044484, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABT6SCSG54XKPKWEQWHC6JLSLHDIZANCNFSM4STHWQWA .
-- Yang Liu Research Scientist Lawrence Berkeley National Laboratory
How many categorical values of the tuning parameter do you have?
There are 10 categorical values, and 9 of them are used in MLA phase, the final one is used in the TLA phase.
Alternatively, can you also try set options['model_class'] = 'Model_GPy_LCM' to see if it works?
I tried, and the program failed as before.
Since I notice that the jacobi matrix is wrong, I realize I can change the config parameters in LBFGS without the jacobi matrix calculated by the program. So I change the command
sol = scipy.optimize.minimize(fun, x0, args=(), method='L-BFGS-B', jac=grad)
to
sol = scipy.optimize.minimize(fun, x0, args=(), method='L-BFGS-B', jac=None)
and it successes ! But the drawback is it will take much much longer time to finsh the MLA phase, which is quite reasonable since the infomation about gradient is missing. Anyway, the program can work now. How do you think ?
I'm glad you found a workaround. It's still somewhat unexpected that jacobi matrix is wrong even using 'Model_GPy_LCM', could you send me your top-level python script (where you define the spaces and call MLA and TLA), as well as the runlog? Please set options['verbose'] = True so that I can see more information.
On Mon, Oct 19, 2020 at 8:10 AM zhf-0 notifications@github.com wrote:
How many categorical values of the tuning parameter do you have?
There are 10 categorical values, and 9 of them are used in MLA phase, the final one is used in the TLA phase.
Alternatively, can you also try set options['model_class'] = 'Model_GPy_LCM' to see if it works?
I tried, and the program failed as before.
Since I notice that the jacobi matrix is wrong, I realize I can change the config parameters in LBFGS without the jacobi matrix calculated by the program. So I change the command
sol = scipy.optimize.minimize(fun, x0, args=(), method='L-BFGS-B', jac=grad)
to
sol = scipy.optimize.minimize(fun, x0, args=(), method='L-BFGS-B', jac=None)
and it successes ! But the drawback is it will take much much longer time to finsh the MLA phase, which is quite reasonable since the infomation about gradient is missing. Anyway, the program can work now. How do you think ?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/gptune/GPTune/issues/1#issuecomment-712232908, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABT6SCVWFYQDNEABSMN4X7DSLRJFRANCNFSM4STHWQWA .
-- Yang Liu Research Scientist Lawrence Berkeley National Laboratory
sure
I see you are treating the number of iterations as floating numbers for the objective, I'm not sure how well GPTune can handle integer valued objective functions. On the other hand, your tuning parameter is threshold = Real (0.1 , 0.9, transform="normalize", name="threshold") which should be easy for GPTune to handle.
Also, try to reduce the number of tasks giventask = [['6'],['7'],['8'],['9'],['10'],['11'],['12'],['13'],['14'],[ '15']] to maybe giventask = [['6']] to see if it works.
If it still doesn't work, you can try the opentuner package, by just setting os.environ['TUNER_NAME'] = 'opentuner'
On Tue, Oct 20, 2020 at 1:40 AM zhf-0 notifications@github.com wrote:
MLA.zip https://github.com/gptune/GPTune/files/5407438/MLA.zip
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/gptune/GPTune/issues/1#issuecomment-712691903, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABT6SCX7WAQRO5VHWY2D5Q3SLVEH7ANCNFSM4STHWQWA .
-- Yang Liu Research Scientist Lawrence Berkeley National Laboratory
BTW, if it's easy to set up your application, we are happy to try to reproduce the error and fix it. The NAN is likely caused by cholesky of a near-singular matrix. So it's possible to tweak it and make the code more robust.
Also, try to reduce the number of tasks giventask = [['6'],['7'],['8'],['9'],['10'],['11'],['12'],['13'],['14'],[ '15']] to maybe giventask = [['6']] to see if it works.
Thanks, I tried and it worked.
BTW, if it's easy to set up your application, we are happy to try to reproduce the error and fix it. The NAN is likely caused by cholesky of a near-singular matrix. So it's possible to tweak it and make the code more robust
That would be great. Recently, I find that sometimes, even when jac=[nan,nan,...]
, the program is still running; but sometimes, it fails. Besides, when jac
is a normal array filled with floating numbers, the program still fails. Maybe, jac
is not the right criterion to determine the failure and success of the program.
The link https://drive.google.com/drive/folders/1KbBDc_athNBpfdkk-UQ-85CqxLDi6vwL?usp=sharing is google drive that contains the data and code you need. The README.md
will explain the usage. Thank you very much.
Thanks for sending the codes. I think the problem is still largely due to that you are treating #of iterations as the objective function. It behaves like a piece-wise constant function (not a perfect function for single GP), which somehow becomes highly correlated across multiple tasks. As a result, using the same number of latent functions as the number of tasks causes over-parameterization and raises trouble for LBFGS.
I tried the example of ntask=10, nrun=40. If I set options['model_latent']<=5, the code works without an issue. Unfortunately, it seems impractical to determine the number of latent functions beforehand. But when such error is to appear again, I added an exception message L-BFGS failed: consider reducing options['model_latent'] ! to remind user to use less latent functions. Please give it a try.
I'm closing this ticket. If you still have issues, feel free to reopen it.
@zhf-0 BTW, we also added an option model_max_jitter_try that controls the number of trials for regularizing the kernel matrix. If I set option['model_max_jitter_try']=15, your code also seems to work without any issue.
PS C:\Users\DELL\Documents\FaceRecog_attendaceSystem-main.final> & C:/Users/DELL/AppData/Local/Programs/Python/Python310/python.exe c:/Users/DELL/Documents/FaceRecog_attendaceSystem-main.final/Face_Recognition.py
Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Users\DELL\AppData\Local\Programs\Python\Python310\lib\tkinter__init.py", line 1921, in call
return self.func(*args)
File "c:\Users\DELL\Documents\FaceRecog_attendaceSystem-main.final\Face_Recognition.py", line 229, in face_recog
img = recognize(img, clf, faceCascade)
File "c:\Users\DELL\Documents\FaceRecog_attendaceSystem-main.final\Face_Recognition.py", line 217, in recognize
coord = draw_boundray(img, faceCascade, 1.1, 10, (255, 255, 255), clf)
File "c:\Users\DELL\Documents\FaceRecog_attendaceSystem-main.final\Face_Recognition.py", line 202, in draw_boundray
self.mark_attendance(i, r, n, d)
File "c:\Users\DELL\Documents\FaceRecog_attendaceSystem-main.final\Face_Recognition.py", line 87, in mark_attendance
uname = n[0:9]
TypeError: 'NoneType' object is not subscriptable
lst=[]
class Face_Recognition:
def init__(self, root):
self.root = root
self.root.geometry("600x450+300+270")
#self.root.iconphoto(False, p1)
#self.root.title("Face_Recognition")
# bg image
# put path of image here
img4 = Image.open(r"myimages\t_btn1.png")
img4 = img4.resize((600, 450), Image.Resampling.LANCZOS)
self.photoimg4 = ImageTk.PhotoImage(img4)
bg_img = Label(self.root, image=self.photoimg4)
bg_img.place(x=0, y=0, width=600, height=450)
# title_lbl = Label(bg_img, text="Mark Attendance ", font=("times new roman", 30, "bold"), fg="black")
#title_lbl.place(x=0, y=0, width=1530, height=200)
#b1_1 = Button(bg_img, text="Click here for attendance!, command=self.face_recog, cursor="hand2",
# font=("times new roman", 18, "bold"), bg="silver", fg="black")
#b1_1.place(x=150, y=320, width=280, height=40)
b1_1 = Button(bg_img, text="Click here for Attendance!",command=self.face_recog, cursor="hand2",
font=("times new roman", 18, "bold"), bg="silver", fg="black")
b1_1.place(x=150, y=320, width=280, height=40)
#img_top = Image.open(r"myimages\face2.png")
#img_top = img_top.resize((300, 300), Image.Resampling.LANCZOS)
#self.photoimg_top = ImageTk.PhotoImage(img_top)
#f_lbl = Label(bg_img, image=self.photoimg_top)
#f_lbl.place(x=700, y=300, width=650, height=400)
#b1_1 = Button(bg_img, text="Click here To Capture !", command=self.face_recog, cursor="hand2",
# font=("times new roman", 18, "bold"), bg="silver", fg="black")
#b1_1.place(x=150, y=300, width=400, height=40)
# save attendance in csv file here
def mark_attendance(self, i, r, n, d):
with open("attendance.csv", "r+", newline="\n") as f:
myDataList = f.readlines()
name_list = []
for line in myDataList:
entry = line.split((","))
name_list.append(entry[0])
if ((i in myDataList) or (r in myDataList) or (n in myDataList) or (d in myDataList)):
pass
elif ((i in myDataList) and (r not in myDataList) and (n not in myDataList) and (d not in myDataList)):
now = datetime.now()
d1 = now.strftime("%d/%m/%Y")
dtString = now.strftime("%H:%M:%S")
f.writelines(f"\n{i[0]},{r},{n},{d},{dtString},{d1},Present")
# speak_va("attendance has been added")
conn = mysql.connector.connect(username='root', password='fazal123', host='localhost', database='face_recognizer1')
mycursor = conn.cursor()
if messagebox.askyesno("Confirmation", "Are you sure you want to save attendance on database?"):
for i in name_list:
uid = i[0]
if r is not None:
uroll = r[0:5]
else:
uroll = ''
uname = n[0:9]
udate = datetime.now().strftime("%Y-%m-%d")
# Check if time-in exists for the student and date
check_qry = "SELECT std_timein, std_timeout FROM stdattendance WHERE std_id=%s and std_date=%s"
mycursor.execute(check_qry, (uid, udate))
result = mycursor.fetchone()
if result is None:
# Insert time-in for the student and date
utimein = datetime.now().strftime("%H:%M:%S")
utimeout = None
uattend = 'present'
insert_qry = "INSERT INTO stdattendance(std_id, std_roll_no, std_name, std_date, std_timein, std_timeout, std_attendance) VALUES(%s,%s,%s,%s,%s,%s,%s)"
mycursor.execute(insert_qry, (uid, uroll, uname, udate, utimein, utimeout, uattend))
elif result[0] is not None and result[1] is None:
# Insert time-out for the student and date
utimeout = datetime.now().strftime("%H:%M:%S")
update_qry = "UPDATE stdattendance SET std_timeout=%s WHERE std_id=%s and std_date=%s"
mycursor.execute(update_qry, (utimeout, uid, udate))
elif result[0] is not None and result[1] is not None:
utimeout = datetime.now().strftime("%H:%M:%S")
update_qry = "UPDATE stdattendance SET std_timeout=%s WHERE std_id=%s and std_date=%s"
mycursor.execute(update_qry, (utimeout, uroll, udate))
# Get the latest time-in time
check_timein_qry = "SELECT MAX(std_timein) FROM stdattendance WHERE std_id=%s and std_date=%s"
mycursor.execute(check_timein_qry, (uroll, udate))
latest_timein = mycursor.fetchone()[0]
# Update the attendance record with the latest time-in time
# update_qry = "UPDATE stdattendance SET std_timein=%s WHERE std_roll_no=%s and std_date=%s and std_timeout=%s"
# mycursor.execute(update_qry, (latest_timein, uroll, udate, utimeout))
# elif result[0] is not None and result[1] is not None and datetime.strptime(result[1], '%H:%M:%S').date() == datetime.now().date():
# Update time-out for the student and date
# utimeout = datetime.now().strftime("%H:%M:%S")
# update_qry = "UPDATE stdattendance SET std_timeout=%s WHERE std_id=%s and std_date=%s"
# mycursor.execute(update_qry, (utimeout, uid, udate))
else:
# Insert time-in for the student and date
utimein = datetime.now().strftime("%H:%M:%S")
utimeout = None
uattend = 'present'
insert_qry = "INSERT INTO stdattendance(std_id, std_roll_no, std_name, std_date, std_timein, std_timeout, std_attendance) VALUES(%s,%s,%s,%s,%s,%s,%s)"
mycursor.execute(insert_qry, (uid, uroll, uname, udate, utimein, utimeout, uattend))
# Fetch all results to prevent "
conn.commit()
conn.close()
def face_recog(self):
# rectangle to focus on face
def draw_boundray(img, classifier, scaleFactor, minNeighbors, color, clf):
gray_image = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
features = classifier.detectMultiScale(gray_image, scaleFactor, minNeighbors)
coord = []
for (x, y, w, h) in features:
cv2.rectangle(img, (x, y), (x + w, y + h), (0, 255, 0), 3)
id, predict = clf.predict(gray_image[y:y + h, x:x + w])
confidence = int((100 * (1 - predict / 300)))
conn = connect(host="localhost", user="root", password="fazal123", database="face_recognizer1")
my_cursor = conn.cursor()
my_cursor.execute("select Name from student where Student_id=" + str(id))
n = my_cursor.fetchone()
try:
n = "+".join(n)
except Exception as e:
pass
my_cursor.execute("select Roll_No from student where Student_id=" + str(id))
r = my_cursor.fetchone()
try:
r = "+".join(r)
except Exception as e:
pass
my_cursor.execute("select Department from student where Student_id=" + str(id))
d = my_cursor.fetchone()
try:
d = "+".join(d)
except Exception as e:
pass
my_cursor.execute("select Student_id from student where Student_id=" + str(id))
i = my_cursor.fetchone()
try:
i = "+".join(i)
except Exception as e:
pass
#recognize face only if confidence is greater than 80%
if confidence > 80:
cv2.putText(img, f"Id :{i}", (x, y - 75), cv2.FONT_HERSHEY_COMPLEX, 0.8, (255, 255, 255), 3)
cv2.putText(img, f"Roll :{r}", (x, y - 55), cv2.FONT_HERSHEY_COMPLEX, 0.8, (255, 255, 255), 3)
cv2.putText(img, f"Name :{n}", (x, y - 30), cv2.FONT_HERSHEY_COMPLEX, 0.8, (255, 255, 255), 3)
cv2.putText(img, f"Department :{d}", (x, y - 5), cv2.FONT_HERSHEY_COMPLEX, 0.8, (255, 255, 255), 3)
if id not in lst:
lst.append(id)
self.mark_attendance(i, r, n, d)
else:
break
else:
cv2.rectangle(img, (x, y), (x + w, y + h), (0, 0, 255), 3)
speak_va("Warning!!! Unknown Face")
cv2.putText(img, "Unknown Face", (x, y - 5), cv2.FONT_HERSHEY_COMPLEX, 0.8, (255, 255, 255), 3)
coord = [x, y, w, y]
return coord
def recognize(img, clf, faceCascade):
coord = draw_boundray(img, faceCascade, 1.1, 10, (255, 255, 255), clf)
if coord is not None:
return img
faceCascade = cv2.CascadeClassifier("haarcascade_frontalface_default.xml")
clf = cv2.face.LBPHFaceRecognizer_create()
clf.read("classifier.xml")
video_cap = cv2.VideoCapture(0)
# increase camera brightness
video_cap.set(cv2.CAP_PROP_BRIGHTNESS, 0.6)
while True:
ret, img = video_cap.read()
img = recognize(img, clf, faceCascade)
cv2.imshow("Welcome", img)
#open camera until enter key is pressed
if cv2.waitKey(1) == 13:
break
video_cap.release()
cv2.destroyAllWindows()
speak_va("thank you")
if name == "main": root = Tk() obj = Face_Recognition(root) root.mainloop() fix the error please
This doesn't seem to be a GPTune related error. Are you using GPTune in your code?
this error is arising becuase the face is not recognised.
This is the wrong place to post. We are doing Bayesian-optimization-based autotuning: https://gptune.lbl.gov/ You are probably using another package with the same name.
hello developers:
Firstly, I want to thank you for offering such a good tool for researching and studying. I have been using it for a while to tackle my own problem and it works great. But recently, a weird error occurs occasionally. When I run the following command
the program did work in the past, but now it fails with the same code. More importantly, when I change the option to
-nrum 40
, it still fails, but when I change it to-nrum 20
, the program works properly!Everytimes the program fails, The error message is the same:
After some basic debug, I find out that when the program fails, in
lcm.py 266
the variableswhich causing the error. Keep backtracking, in the definition of
I suspect that the program jump out of the
if condition
, sohistory_xs = [None]
. But I don't konw how to continue to debug, and come here for help. If needed, I can send all files to you for inspecting. Thank you very much.